2017-08-21 18:48:49,673 - test - INFO - Starting tests with these threads running: '[<_MainThread(MainThread, started 140561159419712)>]'
2017-08-21 18:48:51,282 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 18:48:53,189 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T01:48:51.494457', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 18:48:59,103 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T01:48:51.494457', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 18:48:59,193 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T01:48:51.574211', u'message': u'Validation host configuration lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 18:48:59,227 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T01:48:51.574211', u'message': u'Validation host configuration lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 18:48:59,312 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T01:48:51.629615', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 18:48:59,347 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T01:48:51.629615', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 18:48:59,428 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T01:48:51.687286', u'message': u'Validation host configuration lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 18:48:59,462 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T01:48:51.687286', u'message': u'Validation host configuration lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 18:49:04,395 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/5/'], u'complete': False, u'created_at': u'2017-08-22T01:48:59.752246', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 18:49:30,423 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/5/'], u'complete': True, u'created_at': u'2017-08-22T01:48:59.752246', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 18:49:30,450 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/6/'], u'complete': False, u'created_at': u'2017-08-22T01:49:00.779282', u'message': u'Deploying agent to host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u''}
2017-08-21 18:49:51,255 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/6/'], u'complete': True, u'created_at': u'2017-08-22T01:49:00.779282', u'message': u'Deploying agent to host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u''}
2017-08-21 18:49:51,289 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/7/'], u'complete': False, u'created_at': u'2017-08-22T01:49:01.727541', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u''}
2017-08-21 18:49:54,431 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/7/'], u'complete': True, u'created_at': u'2017-08-22T01:49:01.727541', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u''}
2017-08-21 18:49:54,461 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/8/'], u'complete': False, u'created_at': u'2017-08-22T01:49:02.744373', u'message': u'Deploying agent to host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-21 18:50:01,761 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/8/'], u'complete': True, u'created_at': u'2017-08-22T01:49:02.744373', u'message': u'Deploying agent to host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-21 18:50:02,699 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:03,774 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:04,854 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:05,922 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:06,993 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:08,196 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:09,270 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:10,345 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:11,402 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:12,476 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:13,551 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:14,615 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:15,690 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:16,761 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:17,835 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:18,908 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:19,964 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:21,034 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:22,112 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:23,157 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:24,229 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:25,303 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:26,372 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:27,442 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:28,490 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:29,564 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:30,636 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:31,707 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:32,782 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:33,854 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:34,924 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:35,999 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:37,072 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:38,146 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:39,216 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:40,286 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:41,360 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:42,434 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:43,506 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:44,584 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:45,660 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:46,737 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:47,811 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:48,879 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:49,954 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:51,026 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:52,097 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:53,159 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:54,229 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:55,290 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:56,356 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:57,421 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:58,487 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:50:59,550 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:00,618 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:01,677 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:02,744 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:03,815 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:04,887 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:05,959 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:07,036 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:08,107 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:09,177 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:10,249 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:11,323 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:12,532 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:13,600 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 18:51:15,818 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/'], u'complete': False, u'created_at': u'2017-08-22T01:51:14.096998', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u''}
2017-08-21 19:45:38,919 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/'], u'complete': True, u'created_at': u'2017-08-22T01:51:14.096998', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u'\n\n\nReboot of lotus-59vm15.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-693.1.1.el7.x86_64 to required kernel-3.10.0-514.26.2.el7_lustre.x86_64\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 19:45:38,995 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/', u'/api/job/28/', u'/api/job/29/', u'/api/job/30/'], u'complete': False, u'created_at': u'2017-08-22T01:51:14.358675', u'message': u'Setup managed host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\nReboot of lotus-59vm17.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-693.1.1.el7.x86_64 to required kernel-3.10.0-514.26.2.el7_lustre.x86_64\n'}
2017-08-21 19:48:37,690 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/', u'/api/job/28/', u'/api/job/29/', u'/api/job/30/'], u'complete': True, u'created_at': u'2017-08-22T01:51:14.358675', u'message': u'Setup managed host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\nReboot of lotus-59vm17.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-693.1.1.el7.x86_64 to required kernel-3.10.0-514.26.2.el7_lustre.x86_64\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 19:48:37,752 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/'], u'complete': True, u'created_at': u'2017-08-22T01:51:14.600628', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u'\n\n\nReboot of lotus-59vm16.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-693.1.1.el7.x86_64 to required kernel-3.10.0-514.26.2.el7_lustre.x86_64\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 19:48:37,810 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/'], u'complete': True, u'created_at': u'2017-08-22T01:51:14.600628', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u'\n\n\nReboot of lotus-59vm16.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-693.1.1.el7.x86_64 to required kernel-3.10.0-514.26.2.el7_lustre.x86_64\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 19:48:37,871 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/', u'/api/job/47/', u'/api/job/48/', u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/'], u'complete': False, u'created_at': u'2017-08-22T01:51:14.832564', u'message': u'Setup managed host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n\n\nReboot of lotus-59vm18.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-693.1.1.el7.x86_64 to required kernel-3.10.0-514.26.2.el7_lustre.x86_64\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 19:48:43,449 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/', u'/api/job/47/', u'/api/job/48/', u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/'], u'complete': True, u'created_at': u'2017-08-22T01:51:14.832564', u'message': u'Setup managed host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n\n\nReboot of lotus-59vm18.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-693.1.1.el7.x86_64 to required kernel-3.10.0-514.26.2.el7_lustre.x86_64\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 19:48:44,663 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 19:48:44,663 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:48:45,090 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 19:48:45,090 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:48:45,546 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 19:48:45,546 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:48:45,949 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 19:48:45,950 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:48:47,479 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 19:48:47,480 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 19:48:47,481 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 19:48:47,482 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 19:48:47,849 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 19:48:47,885 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 19:48:47,961 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 19:48:47,967 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 19:48:47,968 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:48:47,969 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:48:47,969 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:48:47,970 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:48:48,337 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 19:48:48,355 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 19:48:48,390 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 19:48:48,448 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 19:48:48,448 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 19:48:49,854 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 19:48:49,855 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:48:50,310 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 19:48:50,311 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:48:50,312 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:48:50,313 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:48:50,313 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:48:50,719 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 19:48:50,834 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 19:48:50,838 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 19:48:50,839 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 19:48:50,840 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 19:48:52,143 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 19:48:52,144 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:48:52,503 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 19:48:52,504 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:48:52,505 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:48:52,506 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:48:52,506 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:48:52,934 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 19:48:53,049 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 19:48:53,056 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 19:48:53,059 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 19:48:53,059 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 19:48:54,472 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 19:48:54,473 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:48:54,822 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 19:48:54,823 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 19:48:55,083 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 
2017-08-21 19:48:55,084 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 19:48:55,411 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 19:48:55,411 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 19:48:56,089 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 19:48:56,090 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 19:48:56,453 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 19:48:56,454 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 19:48:57,087 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 19:48:57,088 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 19:48:57,427 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 19:48:57,428 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 19:48:58,094 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 19:48:58,095 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:48:58,403 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 19:48:58,403 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:48:58,697 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 19:48:58,697 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:48:59,080 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 19:48:59,080 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:48:59,081 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:48:59,442 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_setup_zfs_targets_for_test (tests.integration.shared_storage_configuration.test_aaa_zfs_setup.TestConfigureZfsTargets) ====="
2017-08-21 19:48:59,442 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:48:59,813 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:48:59,813 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:00,216 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_setup_zfs_targets_for_test (tests.integration.shared_storage_configuration.test_aaa_zfs_setup.TestConfigureZfsTargets) ====="
2017-08-21 19:49:00,216 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:00,599 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:49:00,599 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:01,002 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_setup_zfs_targets_for_test (tests.integration.shared_storage_configuration.test_aaa_zfs_setup.TestConfigureZfsTargets) ====="
2017-08-21 19:49:01,002 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:01,378 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:49:01,379 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:01,781 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_setup_zfs_targets_for_test (tests.integration.shared_storage_configuration.test_aaa_zfs_setup.TestConfigureZfsTargets) ====="
2017-08-21 19:49:01,781 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:02,165 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:49:02,165 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:02,877 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_setup_zfs_targets_for_test (tests.integration.shared_storage_configuration.test_aaa_zfs_setup.TestConfigureZfsTargets) ====="
2017-08-21 19:49:02,878 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:03,806 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 19:49:03,806 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:49:03,807 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:04,183 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 19:49:04,183 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 19:49:04,184 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:04,572 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 19:49:04,573 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:49:04,573 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:04,955 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 19:49:04,956 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 19:49:04,956 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:05,357 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 19:49:05,357 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:49:05,358 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:05,759 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 19:49:05,759 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 19:49:05,759 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:06,129 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 19:49:06,130 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:49:06,130 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:06,532 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 19:49:06,533 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 19:49:06,533 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:06,910 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 19:49:06,910 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:49:06,910 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:07,616 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 19:49:07,617 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 19:49:07,617 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:08,309 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 19:49:08,309 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:09,015 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 19:49:09,016 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:09,710 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 19:49:09,710 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:10,373 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 19:49:10,374 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:11,071 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 19:49:11,071 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:11,765 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 19:49:11,766 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:49:12,462 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 19:49:38,084 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 19:49:39,679 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 19:49:41,331 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 19:50:57,837 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 19:50:59,820 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 19:51:00,470 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 19:51:00,470 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 19:51:00,470 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 19:51:04,308 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 19:51:04,308 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:04,692 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 19:51:04,693 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:05,235 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 19:51:05,236 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:05,639 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 19:51:05,640 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:06,181 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 19:51:06,182 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:06,586 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 19:51:06,587 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:07,115 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 19:51:07,116 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:07,515 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 19:51:07,515 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:08,041 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 19:51:08,041 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:08,635 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 19:51:08,636 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:09,034 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which lokkit
2017-08-21 19:51:09,035 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:09,413 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which firewall-cmd
2017-08-21 19:51:09,414 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:09,788 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 19:51:09,789 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:12,621 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 19:51:12,622 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:13,879 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 19:51:13,879 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:15,131 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 19:51:15,132 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:15,506 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 19:51:15,507 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:15,883 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 19:51:15,883 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:16,322 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 19:51:16,323 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:16,715 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 19:51:16,716 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:17,119 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which lokkit
2017-08-21 19:51:17,119 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:17,494 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which firewall-cmd
2017-08-21 19:51:17,494 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:17,867 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 19:51:17,867 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:19,441 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 19:51:19,441 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:20,736 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 19:51:20,736 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:22,005 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 19:51:22,005 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:22,376 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 19:51:22,377 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:22,745 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 19:51:22,746 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:23,176 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 19:51:23,177 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:23,552 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 19:51:23,552 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:23,953 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which lokkit
2017-08-21 19:51:23,953 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:24,325 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which firewall-cmd
2017-08-21 19:51:24,326 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:24,700 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 19:51:24,700 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:27,551 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 19:51:27,551 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:28,898 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 19:51:28,899 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:30,041 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 19:51:30,042 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:30,430 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 19:51:30,430 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:30,804 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 19:51:30,804 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:31,228 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 19:51:31,228 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:31,602 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 19:51:31,602 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:32,002 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which lokkit
2017-08-21 19:51:32,003 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:32,379 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which firewall-cmd
2017-08-21 19:51:32,380 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:32,763 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 19:51:32,763 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:34,357 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 19:51:34,357 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:35,656 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 19:51:35,656 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:36,932 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 19:51:36,933 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:37,307 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 19:51:37,308 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:37,695 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 19:51:37,695 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:38,133 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 19:51:38,133 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:38,523 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 19:51:38,523 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 19:51:38,524 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:38,926 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 19:51:38,927 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:39,328 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 19:51:39,329 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:39,727 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 19:51:39,728 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:40,133 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 19:51:40,133 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:40,839 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:51:40,840 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:41,230 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:51:41,230 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:41,629 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:51:41,629 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:42,033 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:51:42,033 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:42,436 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 19:51:42,436 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:43,141 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 19:51:43,141 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:43,537 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 19:51:43,538 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:43,940 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 19:51:43,941 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:44,343 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 19:51:44,343 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:44,738 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 19:51:44,739 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:45,441 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 19:51:45,441 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:45,846 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 19:51:45,846 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:46,248 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 19:51:46,248 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:46,650 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 19:51:46,651 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:47,054 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 19:51:47,055 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:47,503 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 19:51:47,503 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:47,905 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 19:51:47,906 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:48,302 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 19:51:48,302 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:48,746 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 19:51:48,747 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:51:49,503 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 19:51:49,504 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 19:51:49,505 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 19:51:49,506 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 19:51:49,507 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 19:51:50,099 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:50,100 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:50,110 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:50,121 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:50,303 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:50,304 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 19:51:50,304 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 19:51:50,305 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 19:51:50,305 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 19:51:50,307 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 19:51:50,610 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:50,615 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:50,693 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:50,741 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:51,052 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 19:51:51,053 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 19:51:51,054 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 19:51:51,055 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 19:51:51,056 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 19:51:51,378 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 19:51:51,456 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 19:51:51,564 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 19:51:51,575 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 19:51:51,576 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:51:51,577 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:51:51,577 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:51:51,580 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:51:51,882 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 19:51:51,956 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 19:51:51,992 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 19:51:52,061 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 19:51:52,062 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 19:51:53,379 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 19:51:53,379 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:51:53,929 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 19:51:53,930 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:51:53,931 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:51:53,931 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:51:53,933 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:51:54,299 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 19:51:54,331 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 19:51:54,348 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 19:51:54,398 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 19:51:54,398 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 19:51:56,756 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 19:51:56,757 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:51:57,296 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 19:51:57,297 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:51:57,297 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:51:57,298 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:51:57,298 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:51:57,696 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 19:51:57,742 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 19:51:57,816 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 19:51:57,818 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 19:51:57,819 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 19:52:01,172 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 19:52:01,172 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:52:01,721 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 19:52:01,721 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 19:52:02,004 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 19:52:02,004 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:52:02,318 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 19:52:02,318 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:52:02,633 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 19:52:02,633 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:52:02,928 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 19:52:02,929 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 19:52:03,255 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 19:52:03,255 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 19:52:03,950 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 19:52:03,951 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 19:52:04,271 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 19:52:04,271 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 19:52:04,981 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 19:52:04,982 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 19:52:05,349 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 19:52:05,350 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 19:52:05,922 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 19:52:05,922 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 19:52:06,243 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 19:52:06,243 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 19:52:06,562 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 19:52:06,562 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 19:52:06,860 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 19:52:06,861 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 19:52:06,861 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:52:07,253 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 19:52:07,254 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:52:07,656 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 19:52:07,656 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:52:08,057 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 19:52:08,058 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:52:08,457 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 19:52:08,457 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 19:52:11,405 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 19:52:11,873 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T02:52:10.792250', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 19:52:20,607 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T02:52:10.792250', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 19:52:20,733 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T02:52:10.891950', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 19:52:20,771 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T02:52:10.891950', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 19:52:20,857 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T02:52:10.943889', u'message': u'Validation host configuration lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 19:52:20,889 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T02:52:10.943889', u'message': u'Validation host configuration lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 19:52:20,963 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T02:52:11.011793', u'message': u'Validation host configuration lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 19:52:20,993 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T02:52:11.011793', u'message': u'Validation host configuration lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 19:52:24,812 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/5/'], u'complete': False, u'created_at': u'2017-08-22T02:52:20.938403', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 19:52:30,040 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/5/'], u'complete': True, u'created_at': u'2017-08-22T02:52:20.938403', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 19:52:30,075 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/6/'], u'complete': False, u'created_at': u'2017-08-22T02:52:21.709619', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u''}
2017-08-21 19:52:31,138 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/6/'], u'complete': True, u'created_at': u'2017-08-22T02:52:21.709619', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u''}
2017-08-21 19:52:31,162 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/7/'], u'complete': True, u'created_at': u'2017-08-22T02:52:22.427787', u'message': u'Deploying agent to host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u''}
2017-08-21 19:52:31,184 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/7/'], u'complete': True, u'created_at': u'2017-08-22T02:52:22.427787', u'message': u'Deploying agent to host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u''}
2017-08-21 19:52:31,205 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/8/'], u'complete': False, u'created_at': u'2017-08-22T02:52:23.184377', u'message': u'Deploying agent to host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-21 19:52:33,322 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/8/'], u'complete': True, u'created_at': u'2017-08-22T02:52:23.184377', u'message': u'Deploying agent to host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-21 19:52:34,295 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:35,371 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:36,436 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:37,510 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:38,581 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:39,660 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:40,735 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:41,816 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:42,866 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:43,937 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:45,014 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:46,079 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:47,152 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:48,225 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:49,292 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:50,362 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:51,434 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:52,520 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:53,588 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:54,659 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:55,730 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:56,796 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:57,869 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:52:58,938 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:00,011 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:01,086 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:02,157 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:03,224 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:04,292 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:05,355 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:06,401 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:07,473 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:08,544 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:09,617 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:10,689 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:11,763 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:12,831 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:13,905 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:14,975 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:16,046 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:17,119 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:18,194 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:19,264 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:20,315 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:21,386 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:22,458 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:23,536 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:24,606 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:25,691 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:26,744 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:27,799 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:28,857 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:29,930 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:31,011 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:32,059 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:33,107 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:34,177 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:35,243 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:36,317 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:37,390 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:38,463 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:39,530 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:40,599 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:41,674 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:43,037 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:44,097 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:45,165 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:46,232 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:47,298 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 19:53:49,490 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/'], u'complete': False, u'created_at': u'2017-08-22T02:53:47.752523', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u''}
2017-08-21 20:03:31,723 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/'], u'complete': True, u'created_at': u'2017-08-22T02:53:47.752523', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:03:31,805 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/', u'/api/job/28/', u'/api/job/29/', u'/api/job/30/'], u'complete': True, u'created_at': u'2017-08-22T02:53:48.029898', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:03:31,884 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/', u'/api/job/28/', u'/api/job/29/', u'/api/job/30/'], u'complete': True, u'created_at': u'2017-08-22T02:53:48.029898', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:03:31,949 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/'], u'complete': False, u'created_at': u'2017-08-22T02:53:48.250868', u'message': u'Setup managed host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u'\n\n'}
2017-08-21 20:18:22,648 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/'], u'complete': True, u'created_at': u'2017-08-22T02:53:48.250868', u'message': u'Setup managed host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:18:22,722 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/', u'/api/job/47/', u'/api/job/48/', u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/'], u'complete': True, u'created_at': u'2017-08-22T02:53:48.468413', u'message': u'Setup managed host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:18:22,798 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/', u'/api/job/47/', u'/api/job/48/', u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/'], u'complete': True, u'created_at': u'2017-08-22T02:53:48.468413', u'message': u'Setup managed host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:18:24,022 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:18:24,022 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:24,429 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:18:24,429 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:24,829 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:18:24,829 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:25,222 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:18:25,222 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:26,889 - test - INFO - Configuring power control on [u'lotus-59vm15.lotus.hpdd.lab.intel.com', u'lotus-59vm16.lotus.hpdd.lab.intel.com', u'lotus-59vm17.lotus.hpdd.lab.intel.com', u'lotus-59vm18.lotus.hpdd.lab.intel.com']
2017-08-21 20:18:27,333 - test - DEBUG - Created /api/power_control_type/14/
2017-08-21 20:18:27,429 - test - DEBUG - Created /api/power_control_device/1/
2017-08-21 20:18:28,658 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/1/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm15', u'id': 1, u'resource_uri': u'/api/power_control_device_outlet/1/'}
2017-08-21 20:18:29,694 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/2/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm16', u'id': 2, u'resource_uri': u'/api/power_control_device_outlet/2/'}
2017-08-21 20:18:30,762 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/3/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm17', u'id': 3, u'resource_uri': u'/api/power_control_device_outlet/3/'}
2017-08-21 20:18:31,834 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/4/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm18', u'id': 4, u'resource_uri': u'/api/power_control_device_outlet/4/'}
2017-08-21 20:18:31,911 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm19', u'id': 5, u'resource_uri': u'/api/power_control_device_outlet/5/'}
2017-08-21 20:18:31,911 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 20:18:31,911 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:34,510 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: []
2017-08-21 20:18:34,510 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 20:18:34,511 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:37,114 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com']
2017-08-21 20:18:37,114 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 20:18:37,114 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:39,741 - test - DEBUG - Fencible nodes on lotus-59vm17.lotus.hpdd.lab.intel.com are: ['lotus-59vm17.lotus.hpdd.lab.intel.com']
2017-08-21 20:18:39,741 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 20:18:39,742 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:42,345 - test - DEBUG - Fencible nodes on lotus-59vm18.lotus.hpdd.lab.intel.com are: ['lotus-59vm18.lotus.hpdd.lab.intel.com', 'lotus-59vm17.lotus.hpdd.lab.intel.com']
2017-08-21 20:18:43,346 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 20:18:43,346 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:45,985 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 20:18:45,985 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 20:18:45,985 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:48,652 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 20:18:48,653 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 20:18:48,653 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:51,265 - test - DEBUG - Fencible nodes on lotus-59vm17.lotus.hpdd.lab.intel.com are: ['lotus-59vm18.lotus.hpdd.lab.intel.com', 'lotus-59vm17.lotus.hpdd.lab.intel.com']
2017-08-21 20:18:51,266 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 20:18:51,268 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:18:53,834 - test - DEBUG - Fencible nodes on lotus-59vm18.lotus.hpdd.lab.intel.com are: ['lotus-59vm18.lotus.hpdd.lab.intel.com', 'lotus-59vm17.lotus.hpdd.lab.intel.com']
2017-08-21 20:18:55,733 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/47/', u'volume_nodes': [{u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 3, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/21/', u'id': 21, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 4, u'resource_uri': u'/api/volume_node/22/', u'id': 22, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/26/', u'volume_nodes': [{u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 3, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/7/', u'id': 7, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 4, u'resource_uri': u'/api/volume_node/12/', u'id': 12, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/17/', u'id': 17, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/49/', u'volume_nodes': [{u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 3, u'resource_uri': u'/api/volume_node/4/', u'id': 4, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 4, u'resource_uri': u'/api/volume_node/14/', u'id': 14, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/19/', u'id': 19, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/23/', u'volume_nodes': [{u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 3, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/6/', u'id': 6, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 4, u'resource_uri': u'/api/volume_node/11/', u'id': 11, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/16/', u'id': 16, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/51/', u'volume_nodes': [{u'use': True, u'volume_id': 5, u'primary': True, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 3, u'resource_uri': u'/api/volume_node/5/', u'id': 5, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/10/', u'id': 10, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 4, u'resource_uri': u'/api/volume_node/15/', u'id': 15, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/20/', u'id': 20, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/5/', u'id': 5, u'size': u'10672993730'}]'
2017-08-21 20:18:55,734 - test - DEBUG -         self.wait_until_true(lambda: len(self.get_shared_volumes(required_hosts)) >= expected_volumes)
 evaluated to True
2017-08-21 20:18:57,461 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/47/', u'volume_nodes': [{u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 3, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/21/', u'id': 21, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 4, u'resource_uri': u'/api/volume_node/22/', u'id': 22, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/26/', u'volume_nodes': [{u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 3, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/7/', u'id': 7, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 4, u'resource_uri': u'/api/volume_node/12/', u'id': 12, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/17/', u'id': 17, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/49/', u'volume_nodes': [{u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 3, u'resource_uri': u'/api/volume_node/4/', u'id': 4, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 4, u'resource_uri': u'/api/volume_node/14/', u'id': 14, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/19/', u'id': 19, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/23/', u'volume_nodes': [{u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 3, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/6/', u'id': 6, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 4, u'resource_uri': u'/api/volume_node/11/', u'id': 11, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/16/', u'id': 16, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/51/', u'volume_nodes': [{u'use': True, u'volume_id': 5, u'primary': True, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 3, u'resource_uri': u'/api/volume_node/5/', u'id': 5, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/10/', u'id': 10, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 4, u'resource_uri': u'/api/volume_node/15/', u'id': 15, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/20/', u'id': 20, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/5/', u'id': 5, u'size': u'10672993730'}]'
2017-08-21 20:18:59,242 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/59/', u'/api/job/60/', u'/api/job/61/', u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/', u'/api/job/70/', u'/api/job/71/', u'/api/job/72/', u'/api/job/73/', u'/api/job/74/', u'/api/job/75/'], u'complete': False, u'created_at': u'2017-08-22T03:18:57.491732', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/19/', u'id': 19, u'logs': u'\n\n\n'}
2017-08-21 20:22:54,619 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/59/', u'/api/job/60/', u'/api/job/61/', u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/', u'/api/job/70/', u'/api/job/71/', u'/api/job/72/', u'/api/job/73/', u'/api/job/74/', u'/api/job/75/'], u'complete': True, u'created_at': u'2017-08-22T03:18:57.491732', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/19/', u'id': 19, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:22:54,623 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 20:22:54,623 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:22:55,029 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 20:22:55,029 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:22:55,505 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 20:22:55,505 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:22:55,921 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 20:22:55,921 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:01,658 - test - DEBUG - MGS: should be running on lotus-59vm15.lotus.hpdd.lab.intel.com (actual: lotus-59vm15.lotus.hpdd.lab.intel.com)
2017-08-21 20:23:01,659 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: crm_resource -r MGS_0aa2e9 -W
2017-08-21 20:23:01,659 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:02,109 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-MGS_0aa2e9 -N lotus-59vm15.lotus.hpdd.lab.intel.com -G -d 0
2017-08-21 20:23:02,109 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:02,532 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-21 20:23:02,533 - test - DEBUG - testfs-MDT0000: should be running on lotus-59vm16.lotus.hpdd.lab.intel.com (actual: lotus-59vm16.lotus.hpdd.lab.intel.com)
2017-08-21 20:23:02,533 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: crm_resource -r testfs-MDT0000_8b2be3 -W
2017-08-21 20:23:02,533 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:02,936 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-testfs-MDT0000_8b2be3 -N lotus-59vm16.lotus.hpdd.lab.intel.com -G -d 0
2017-08-21 20:23:02,937 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:03,337 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-21 20:23:03,337 - test - DEBUG - testfs-OST0000: should be running on lotus-59vm17.lotus.hpdd.lab.intel.com (actual: lotus-59vm17.lotus.hpdd.lab.intel.com)
2017-08-21 20:23:03,337 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: crm_resource -r testfs-OST0000_1cea31 -W
2017-08-21 20:23:03,337 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:03,764 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-testfs-OST0000_1cea31 -N lotus-59vm17.lotus.hpdd.lab.intel.com -G -d 0
2017-08-21 20:23:03,764 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:04,137 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-21 20:23:04,137 - test - DEBUG - testfs-OST0001: should be running on lotus-59vm18.lotus.hpdd.lab.intel.com (actual: lotus-59vm18.lotus.hpdd.lab.intel.com)
2017-08-21 20:23:04,137 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: crm_resource -r testfs-OST0001_9e620d -W
2017-08-21 20:23:04,138 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:04,541 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-testfs-OST0001_9e620d -N lotus-59vm18.lotus.hpdd.lab.intel.com -G -d 0
2017-08-21 20:23:04,541 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:04,941 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-21 20:23:05,617 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent stop_target --ha MGS_0aa2e9
2017-08-21 20:23:05,617 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:24,431 - test - DEBUG -             self.wait_for_assert(lambda: self.assertHasAlert(mgt['resource_uri'], of_severity='ERROR'))
 tripped assertion: u'/api/target/1/' not found in []
2017-08-21 20:23:25,462 - test - DEBUG -             self.wait_for_assert(lambda: self.assertHasAlert(mgt['resource_uri'], of_severity='ERROR'))
 tripped assertion: u'/api/target/1/' not found in []
2017-08-21 20:23:26,497 - test - DEBUG -             self.wait_for_assert(lambda: self.assertHasAlert(mgt['resource_uri'], of_severity='ERROR'))
 tripped assertion: u'/api/target/1/' not found in []
2017-08-21 20:23:27,531 - test - DEBUG -             self.wait_for_assert(lambda: self.assertHasAlert(mgt['resource_uri'], of_severity='ERROR'))
 tripped assertion: u'/api/target/1/' not found in []
2017-08-21 20:23:28,577 - test - DEBUG - assertValue state /api/target/1/ unmounted
2017-08-21 20:23:28,721 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent start_target --ha MGS_0aa2e9
2017-08-21 20:23:28,721 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:23:43,628 - test - DEBUG -         self.wait_for_assert(lambda: self.assertNoAlerts(mgt['resource_uri']))
 tripped assertion: u'/api/target/1/' unexpectedly found in [u'/api/target/1/']
2017-08-21 20:23:44,669 - test - DEBUG -         self.wait_for_assert(lambda: self.assertNoAlerts(mgt['resource_uri']))
 tripped assertion: u'/api/target/1/' unexpectedly found in [u'/api/target/1/']
2017-08-21 20:23:45,713 - test - DEBUG -         self.wait_for_assert(lambda: self.assertNoAlerts(mgt['resource_uri']))
 tripped assertion: u'/api/target/1/' unexpectedly found in [u'/api/target/1/']
2017-08-21 20:23:46,752 - test - DEBUG -         self.wait_for_assert(lambda: self.assertNoAlerts(mgt['resource_uri']))
 tripped assertion: u'/api/target/1/' unexpectedly found in [u'/api/target/1/']
2017-08-21 20:23:47,790 - test - DEBUG -         self.wait_for_assert(lambda: self.assertNoAlerts(mgt['resource_uri']))
 tripped assertion: u'/api/target/1/' unexpectedly found in [u'/api/target/1/']
2017-08-21 20:23:48,829 - test - DEBUG -         self.wait_for_assert(lambda: self.assertNoAlerts(mgt['resource_uri']))
 tripped assertion: u'/api/target/1/' unexpectedly found in [u'/api/target/1/']
2017-08-21 20:23:49,874 - test - DEBUG -         self.wait_for_assert(lambda: self.assertNoAlerts(mgt['resource_uri']))
 tripped assertion: u'/api/target/1/' unexpectedly found in [u'/api/target/1/']
2017-08-21 20:23:50,908 - test - DEBUG - set_state /api/target/1/ unmounted
2017-08-21 20:23:51,221 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/80/'], u'complete': False, u'created_at': u'2017-08-22T03:23:50.366286', u'message': u'Stop target MGS', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/24/', u'id': 24, u'logs': u''}
2017-08-21 20:24:18,369 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/80/'], u'complete': True, u'created_at': u'2017-08-22T03:23:50.366286', u'message': u'Stop target MGS', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/24/', u'id': 24, u'logs': u''}
2017-08-21 20:24:18,369 - test - DEBUG - assertValue state /api/target/1/ unmounted
2017-08-21 20:24:18,552 - test - DEBUG - set_state /api/filesystem/1/ stopped
2017-08-21 20:24:19,014 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/81/', u'/api/job/82/', u'/api/job/83/', u'/api/job/84/'], u'complete': False, u'created_at': u'2017-08-22T03:24:18.097171', u'message': u'Stop file system testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/25/', u'id': 25, u'logs': u''}
2017-08-21 20:24:48,637 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/81/', u'/api/job/82/', u'/api/job/83/', u'/api/job/84/'], u'complete': True, u'created_at': u'2017-08-22T03:24:18.097171', u'message': u'Stop file system testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/25/', u'id': 25, u'logs': u'\n\n'}
2017-08-21 20:24:48,638 - test - DEBUG - assertValue state /api/filesystem/1/ stopped
2017-08-21 20:24:49,237 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent stop_lnet
2017-08-21 20:24:49,237 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:25:06,139 - test - DEBUG -         self.wait_for_assert(lambda: self.assertHasAlert(host['lnet_configuration'], of_severity='INFO'))
 tripped assertion: u'/api/lnet_configuration/1/' not found in []
2017-08-21 20:25:07,174 - test - DEBUG -         self.wait_for_assert(lambda: self.assertHasAlert(host['lnet_configuration'], of_severity='INFO'))
 tripped assertion: u'/api/lnet_configuration/1/' not found in []
2017-08-21 20:25:08,208 - test - DEBUG -         self.wait_for_assert(lambda: self.assertHasAlert(host['lnet_configuration'], of_severity='INFO'))
 tripped assertion: u'/api/lnet_configuration/1/' not found in []
2017-08-21 20:25:09,243 - test - DEBUG -         self.wait_for_assert(lambda: self.assertHasAlert(host['lnet_configuration'], of_severity='INFO'))
 tripped assertion: u'/api/lnet_configuration/1/' not found in []
2017-08-21 20:25:10,289 - test - DEBUG - assertValue state /api/lnet_configuration/1/ lnet_down
2017-08-21 20:25:10,482 - test - DEBUG - set_state /api/lnet_configuration/1/ lnet_up
2017-08-21 20:25:10,747 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/85/'], u'complete': False, u'created_at': u'2017-08-22T03:25:09.879402', u'message': u'Start the LNet networking layer.', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/26/', u'id': 26, u'logs': u''}
2017-08-21 20:25:13,900 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/85/'], u'complete': True, u'created_at': u'2017-08-22T03:25:09.879402', u'message': u'Start the LNet networking layer.', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/26/', u'id': 26, u'logs': u'\n'}
2017-08-21 20:25:13,900 - test - DEBUG - assertValue state /api/lnet_configuration/1/ lnet_up
2017-08-21 20:25:14,039 - test - DEBUG - set_state /api/lnet_configuration/1/ lnet_down
2017-08-21 20:25:14,366 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/86/'], u'complete': False, u'created_at': u'2017-08-22T03:25:13.492240', u'message': u'Shut down the LNet networking layer and stop any targets running on this server.', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/27/', u'id': 27, u'logs': u''}
2017-08-21 20:25:28,058 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/86/'], u'complete': True, u'created_at': u'2017-08-22T03:25:13.492240', u'message': u'Shut down the LNet networking layer and stop any targets running on this server.', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/27/', u'id': 27, u'logs': u'\n'}
2017-08-21 20:25:28,059 - test - DEBUG - assertValue state /api/lnet_configuration/1/ lnet_down
2017-08-21 20:25:28,171 - test - DEBUG - set_state /api/filesystem/1/ available
2017-08-21 20:25:28,689 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/87/', u'/api/job/88/', u'/api/job/89/', u'/api/job/90/', u'/api/job/91/', u'/api/job/92/'], u'complete': False, u'created_at': u'2017-08-22T03:25:27.723053', u'message': u'Start file system testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/28/', u'id': 28, u'logs': u''}
2017-08-21 20:27:57,166 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/87/', u'/api/job/88/', u'/api/job/89/', u'/api/job/90/', u'/api/job/91/', u'/api/job/92/'], u'complete': True, u'created_at': u'2017-08-22T03:25:27.723053', u'message': u'Start file system testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/28/', u'id': 28, u'logs': u'\n\n\n\n\n'}
2017-08-21 20:27:57,166 - test - DEBUG - assertValue state /api/filesystem/1/ available
2017-08-21 20:27:58,175 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent stop_target --ha MGS_0aa2e9
2017-08-21 20:27:58,176 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:28:27,432 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: chroma-agent stop_target --ha testfs-MDT0000_8b2be3
2017-08-21 20:28:27,433 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:28:38,446 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: chroma-agent stop_target --ha testfs-OST0000_1cea31
2017-08-21 20:28:38,446 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:28:57,616 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: chroma-agent stop_target --ha testfs-OST0001_9e620d
2017-08-21 20:28:57,616 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:29:04,129 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent stop_lnet
2017-08-21 20:29:04,130 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:29:21,004 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent stop_pacemaker
2017-08-21 20:29:21,005 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:29:28,598 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent stop_corosync
2017-08-21 20:29:28,598 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:29:40,475 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent start_lnet
2017-08-21 20:29:40,475 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:29:43,233 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent start_corosync
2017-08-21 20:29:43,233 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:29:50,967 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-agent start_pacemaker
2017-08-21 20:29:50,968 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:30:00,975 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:30:00,975 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:30:01,687 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:30:01,688 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:30:02,378 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:30:02,378 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:30:03,071 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:30:03,072 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:30:03,783 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:30:03,783 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:30:04,498 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:30:04,499 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:30:05,484 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/93/', u'/api/job/94/', u'/api/job/95/', u'/api/job/96/', u'/api/job/97/'], u'complete': False, u'created_at': u'2017-08-22T03:30:04.493606', u'message': u'Remove file system testfs from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/29/', u'id': 29, u'logs': u''}
2017-08-21 20:30:46,201 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/93/', u'/api/job/94/', u'/api/job/95/', u'/api/job/96/', u'/api/job/97/'], u'complete': True, u'created_at': u'2017-08-22T03:30:04.493606', u'message': u'Remove file system testfs from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/29/', u'id': 29, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:30:46,527 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/98/', u'/api/job/99/'], u'complete': False, u'created_at': u'2017-08-22T03:30:45.652038', u'message': u'Remove target MGS from the filesystem\n*** WARNING *** this is destructive to your filesystem and irreversible', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/30/', u'id': 30, u'logs': u''}
2017-08-21 20:31:43,184 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/98/', u'/api/job/99/'], u'complete': True, u'created_at': u'2017-08-22T03:30:45.652038', u'message': u'Remove target MGS from the filesystem\n*** WARNING *** this is destructive to your filesystem and irreversible', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/30/', u'id': 30, u'logs': u'\n\n\n\n'}
2017-08-21 20:31:45,702 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/100/', u'/api/job/101/', u'/api/job/102/', u'/api/job/103/', u'/api/job/104/', u'/api/job/105/', u'/api/job/106/', u'/api/job/107/', u'/api/job/108/'], u'complete': False, u'created_at': u'2017-08-22T03:31:43.920209', u'message': u'Remove host lotus-59vm15.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/31/', u'id': 31, u'logs': u'\n\n\n'}
2017-08-21 20:32:11,608 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/100/', u'/api/job/101/', u'/api/job/102/', u'/api/job/103/', u'/api/job/104/', u'/api/job/105/', u'/api/job/106/', u'/api/job/107/', u'/api/job/108/'], u'complete': True, u'created_at': u'2017-08-22T03:31:43.920209', u'message': u'Remove host lotus-59vm15.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/31/', u'id': 31, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:32:11,670 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/109/', u'/api/job/110/', u'/api/job/111/', u'/api/job/112/', u'/api/job/113/', u'/api/job/114/', u'/api/job/115/', u'/api/job/116/', u'/api/job/117/'], u'complete': False, u'created_at': u'2017-08-22T03:31:44.228472', u'message': u'Remove host lotus-59vm16.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/32/', u'id': 32, u'logs': u'\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:32:20,373 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/109/', u'/api/job/110/', u'/api/job/111/', u'/api/job/112/', u'/api/job/113/', u'/api/job/114/', u'/api/job/115/', u'/api/job/116/', u'/api/job/117/'], u'complete': True, u'created_at': u'2017-08-22T03:31:44.228472', u'message': u'Remove host lotus-59vm16.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/32/', u'id': 32, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:32:20,450 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/118/', u'/api/job/119/', u'/api/job/120/', u'/api/job/121/', u'/api/job/122/', u'/api/job/123/', u'/api/job/124/', u'/api/job/125/', u'/api/job/126/'], u'complete': True, u'created_at': u'2017-08-22T03:31:44.452940', u'message': u'Remove host lotus-59vm17.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/33/', u'id': 33, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:32:20,529 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/118/', u'/api/job/119/', u'/api/job/120/', u'/api/job/121/', u'/api/job/122/', u'/api/job/123/', u'/api/job/124/', u'/api/job/125/', u'/api/job/126/'], u'complete': True, u'created_at': u'2017-08-22T03:31:44.452940', u'message': u'Remove host lotus-59vm17.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/33/', u'id': 33, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:32:20,591 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/127/', u'/api/job/128/', u'/api/job/129/', u'/api/job/130/', u'/api/job/131/', u'/api/job/132/', u'/api/job/133/', u'/api/job/134/', u'/api/job/135/'], u'complete': False, u'created_at': u'2017-08-22T03:31:44.727065', u'message': u'Remove host lotus-59vm18.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/34/', u'id': 34, u'logs': u'\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:32:21,724 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/127/', u'/api/job/128/', u'/api/job/129/', u'/api/job/130/', u'/api/job/131/', u'/api/job/132/', u'/api/job/133/', u'/api/job/134/', u'/api/job/135/'], u'complete': True, u'created_at': u'2017-08-22T03:31:44.727065', u'message': u'Remove host lotus-59vm18.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/34/', u'id': 34, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:32:21,902 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:32:21,902 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:22,304 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 20:32:22,304 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:22,707 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:32:22,708 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:23,103 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 20:32:23,103 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:23,481 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:32:23,481 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:23,883 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 20:32:23,883 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:24,257 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:32:24,258 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:24,661 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 20:32:24,661 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:25,035 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:32:25,035 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:25,749 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_alerts (tests.integration.shared_storage_configuration.test_alerting.TestAlerting) ====="
2017-08-21 20:32:25,750 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:26,820 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:32:26,820 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:32:26,820 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:27,195 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 20:32:27,195 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_reboot_event (tests.integration.shared_storage_configuration.test_alerting.TestEvents) ====="
2017-08-21 20:32:27,196 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:27,590 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:32:27,590 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:28,328 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:32:28,328 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:29,026 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:32:29,026 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:29,723 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:32:29,724 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:30,433 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:32:30,434 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:31,131 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:32:31,132 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:32:31,825 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 20:33:01,171 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 20:33:02,801 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 20:33:04,462 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 20:34:20,410 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 20:34:22,313 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 20:34:22,842 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 20:34:22,842 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 20:34:22,843 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 20:34:26,729 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:34:26,729 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:27,133 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:34:27,133 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:27,564 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:34:27,564 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:27,966 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:34:27,966 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:28,392 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:34:28,392 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:28,791 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:34:28,791 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:29,215 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:34:29,216 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:29,616 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:34:29,616 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:30,051 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:34:30,051 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:30,759 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:34:30,759 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:31,163 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:34:31,163 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:31,703 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:34:31,704 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:33,005 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:34:33,006 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:34,375 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:34:34,375 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:34,752 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:34:34,752 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:35,194 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:34:35,194 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:35,646 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:34:35,646 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:36,048 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:34:36,048 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:36,549 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:34:36,549 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:36,926 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:34:36,926 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:37,310 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:34:37,311 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:37,693 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:34:37,694 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:38,079 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:34:38,484 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:34:38,485 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:34:38,869 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:34:38,870 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:34:38,870 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:34:38,872 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:34:38,873 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:34:39,280 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:34:39,281 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:34:39,398 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:34:39,401 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:34:39,402 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:34:39,402 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:34:39,403 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:34:39,403 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:34:39,699 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:34:39,767 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:34:39,802 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:34:39,826 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:34:39,826 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:34:41,294 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 20:34:41,294 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:34:41,918 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:34:41,918 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:34:41,919 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:34:41,920 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:34:41,922 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:34:42,280 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:34:42,309 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:34:42,364 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:34:42,430 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:34:42,430 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:34:44,805 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 20:34:44,805 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:34:45,455 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:34:45,456 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:34:45,457 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:34:45,458 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:34:45,459 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:34:45,774 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:34:45,854 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:34:45,956 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:34:45,959 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:34:45,959 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:34:49,458 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 20:34:49,458 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:34:50,066 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:34:50,066 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 20:34:50,345 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/MGS
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-OST0000

2017-08-21 20:34:50,346 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-OST0000
2017-08-21 20:34:50,657 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-OST0000) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-OST0000 exit_status 0 output:
 
2017-08-21 20:34:50,658 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/MGS
2017-08-21 20:34:51,106 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/MGS) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/MGS exit_status 0 output:
 
2017-08-21 20:34:51,106 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:34:51,406 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:34:51,407 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:34:51,724 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:34:51,725 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:34:52,022 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:34:52,023 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:34:52,477 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:34:52,477 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 20:34:52,998 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 20:34:52,999 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:34:53,479 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:34:53,480 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 20:34:54,129 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 20:34:54,130 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:34:54,579 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:34:54,580 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 20:34:55,240 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 20:34:55,241 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:34:55,547 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:34:55,547 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:34:55,964 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:34:55,965 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:34:56,245 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:34:56,245 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:34:56,245 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:34:58,233 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 20:34:59,994 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T03:34:58.366321', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 20:35:08,082 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T03:34:58.366321', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 20:35:09,040 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': False, u'created_at': u'2017-08-22T03:35:07.996752', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 20:35:18,431 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T03:35:07.996752', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 20:35:18,907 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:19,954 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:20,997 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:22,039 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:23,081 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:24,130 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:25,171 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:26,213 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:27,252 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:28,292 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:29,345 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:30,386 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:31,433 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:32,471 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:33,513 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:34,551 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:35,591 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:36,634 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:37,675 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:38,715 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:39,753 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:40,799 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:41,841 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:42,883 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:43,923 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:44,964 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:46,010 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:47,052 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:48,093 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:49,132 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:50,174 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:51,211 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:52,250 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:53,288 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:54,330 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:55,371 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:56,410 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:57,451 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:58,496 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:35:59,536 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:00,578 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:01,617 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:02,657 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:03,697 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:04,737 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:05,777 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:06,821 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:07,861 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:08,896 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:09,940 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:10,981 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:12,022 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:13,064 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:14,110 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:15,143 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:16,185 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:17,225 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:18,266 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:19,308 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:20,349 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:21,392 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:22,434 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:23,475 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:24,516 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:25,557 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:26,595 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:27,639 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:28,686 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:29,737 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:30,777 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:31,819 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:32,862 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:33,904 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:34,944 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 20:36:36,482 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': False, u'created_at': u'2017-08-22T03:36:35.386117', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 20:47:24,604 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': True, u'created_at': u'2017-08-22T03:36:35.386117', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 20:47:25,018 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:47:25,018 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:47:26,135 - test - DEBUG - remote_command[lotus-59]: [ $(virsh domstate lotus-59vm15) = shut\ off ] || virsh destroy lotus-59vm15
2017-08-21 20:47:26,135 - test - INFO - SSH address = lotus-59, args = {'username': 'root'}
2017-08-21 20:47:28,014 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:47:28,014 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:48:34,144 - test - DEBUG - Unknown socket error when checking lotus-59vm15.lotus.hpdd.lab.intel.com
2017-08-21 20:48:34,145 - test - DEBUG - remote_command[lotus-59]: virsh start lotus-59vm15 || [ $(virsh domstate lotus-59vm15) = running ]
2017-08-21 20:48:34,145 - test - INFO - SSH address = lotus-59, args = {'username': 'root'}
2017-08-21 20:48:35,904 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com started successfully
2017-08-21 20:48:35,905 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:48:35,905 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:48:38,912 - test - DEBUG - Unknown socket error when checking lotus-59vm15.lotus.hpdd.lab.intel.com
2017-08-21 20:48:41,916 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:48:41,916 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:48:44,924 - test - DEBUG - Unknown socket error when checking lotus-59vm15.lotus.hpdd.lab.intel.com
2017-08-21 20:48:47,927 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:48:47,928 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:48:50,938 - test - DEBUG - Unknown socket error when checking lotus-59vm15.lotus.hpdd.lab.intel.com
2017-08-21 20:48:53,941 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:48:53,942 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:48:54,512 - test - DEBUG - Unknown socket error when checking lotus-59vm15.lotus.hpdd.lab.intel.com
2017-08-21 20:48:57,516 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:48:57,516 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:48:57,519 - test - DEBUG - Unknown socket error when checking lotus-59vm15.lotus.hpdd.lab.intel.com
2017-08-21 20:49:00,523 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:49:00,523 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:01,158 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:01,216 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:01,420 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:02,463 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:03,503 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:04,545 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:05,584 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:06,623 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:07,677 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:08,731 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:09,773 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:10,833 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:11,878 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:12,922 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:13,967 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:15,007 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:16,047 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:17,091 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:18,134 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:19,178 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:20,215 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:21,257 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:22,297 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:23,337 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:24,381 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:25,425 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:26,466 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:27,510 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:28,552 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to False
2017-08-21 20:49:29,605 - test - DEBUG -         def reboot_event_was_seen():
            events = self.get_list("/api/alert/", {'begin__gte': start_time})

            reboot_events = [e for e in events if e['message'].find("restarted") != -1]
            return len(reboot_events) == 1
 evaluated to True
2017-08-21 20:49:29,606 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:49:29,606 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:29,992 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_reboot_event (tests.integration.shared_storage_configuration.test_alerting.TestEvents) ====="
2017-08-21 20:49:29,992 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:30,465 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:49:30,466 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:49:30,466 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:30,900 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 20:49:30,901 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:49:30,901 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:31,280 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:49:31,280 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:49:31,280 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:31,682 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 20:49:31,682 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:49:31,682 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:32,068 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:49:32,068 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:49:32,069 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:32,467 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 20:49:32,467 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:49:32,468 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:32,843 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:49:32,843 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:49:32,843 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:33,246 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 20:49:33,246 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:49:33,246 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:33,620 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:49:33,620 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:49:33,621 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:34,330 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 20:49:34,330 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:49:34,330 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:35,029 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:49:35,029 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:35,736 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:49:35,737 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:36,438 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:49:36,438 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:37,136 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:49:37,136 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:37,835 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:49:37,835 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:38,546 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:49:38,547 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:49:39,242 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 20:50:06,520 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 20:50:08,244 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 20:50:10,052 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 20:51:27,856 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 20:51:29,684 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 20:51:30,325 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 20:51:30,326 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 20:51:30,326 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 20:51:34,219 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:51:34,219 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:34,582 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:51:34,582 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:35,111 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:51:35,111 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:35,512 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:51:35,512 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:35,893 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:51:35,893 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:36,287 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:51:36,287 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:36,662 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:51:36,662 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:37,063 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:51:37,064 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:37,452 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:51:37,452 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:38,157 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:51:38,158 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:38,536 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:51:38,536 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:40,120 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:51:40,120 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:41,327 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:51:41,327 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:42,624 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:51:42,625 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:43,001 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:51:43,001 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:43,378 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:51:43,378 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:43,812 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:51:43,812 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:44,186 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:51:44,186 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:44,585 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:51:44,585 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:45,111 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:51:45,111 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:46,366 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:51:46,367 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:47,667 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:51:47,667 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:48,040 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:51:48,041 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:48,412 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:51:48,412 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:48,811 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:51:48,812 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:49,183 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:51:49,183 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:49,585 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:51:49,586 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:50,107 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:51:50,107 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:51,365 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:51:51,366 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:52,623 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:51:52,624 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:53,019 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:51:53,020 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:53,407 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:51:53,407 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:53,831 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:51:53,831 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:54,210 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:51:54,210 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:54,612 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:51:54,612 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:55,140 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:51:55,140 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:56,389 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:51:56,390 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:57,644 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:51:57,645 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:58,025 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:51:58,025 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:58,414 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:51:58,414 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:58,851 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:51:58,851 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:59,234 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 20:51:59,234 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:51:59,234 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:51:59,637 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:51:59,638 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:00,041 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:52:00,042 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:00,442 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:52:00,442 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:00,846 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:52:00,846 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:01,547 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:52:01,548 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:01,951 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:52:01,952 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:02,353 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:52:02,353 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:02,727 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:52:02,727 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:03,127 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:52:03,128 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:03,829 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:52:03,830 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:04,228 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:52:04,229 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:04,629 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:52:04,630 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:05,021 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:52:05,021 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:05,419 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:52:05,419 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:06,126 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:52:06,127 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:06,530 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:52:06,530 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:06,932 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:52:06,933 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:07,331 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:52:07,331 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:07,728 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:52:07,728 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:08,104 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:52:08,105 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:08,503 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:52:08,503 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:08,898 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:52:08,899 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:09,299 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:52:09,299 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:10,052 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:52:10,053 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:52:10,054 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:52:10,054 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:52:10,055 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:52:10,453 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:10,501 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:10,638 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:10,641 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:10,795 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:10,796 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:52:10,797 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:52:10,798 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:52:10,799 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:52:10,803 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:52:11,101 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:11,130 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:11,310 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:11,324 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:11,334 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:52:11,336 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:52:11,337 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:52:11,338 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:52:11,339 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:52:11,670 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:52:11,745 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:52:11,844 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:52:11,862 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:52:11,863 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:52:11,864 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:52:11,864 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:52:11,865 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:52:12,174 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:52:12,181 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:52:12,243 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:52:12,292 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:52:12,293 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:52:13,649 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 20:52:13,649 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:52:14,280 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:52:14,281 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:52:14,282 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:52:14,282 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:52:14,284 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:52:14,628 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:52:14,705 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:52:14,748 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:52:14,824 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:52:14,825 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:52:17,297 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 20:52:17,297 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:52:17,798 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:52:17,798 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:52:17,800 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:52:17,801 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:52:17,804 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:52:18,127 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:52:18,130 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:52:18,200 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:52:18,311 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:52:18,312 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:52:21,782 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 20:52:21,782 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:52:22,276 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:52:22,276 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 20:52:22,634 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 20:52:22,634 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:52:22,930 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:52:22,931 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:52:23,333 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:52:23,333 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:52:23,735 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:52:23,736 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:52:24,080 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:52:24,081 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 20:52:24,621 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 20:52:24,622 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:52:25,064 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:52:25,065 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 20:52:25,600 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 20:52:25,600 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:52:25,956 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:52:25,956 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 20:52:26,506 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 20:52:26,507 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:52:26,944 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:52:26,944 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:52:27,347 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:52:27,347 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:52:27,629 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:52:27,629 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:52:27,629 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:28,015 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:52:28,015 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:28,405 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:52:28,405 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:28,807 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:52:28,807 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:29,205 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:52:29,206 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:32,190 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 20:52:35,791 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: echo ALLOW_ANONYMOUS_READ = False > /usr/share/chroma-manager/local_settings.py
2017-08-21 20:52:35,792 - test - INFO - SSH address = lotus-59vm13.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:52:36,491 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config restart
2017-08-21 20:52:36,491 - test - INFO - SSH address = lotus-59vm13.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:53:18,772 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: rm -rf /usr/share/chroma-manager/local_settings.py*
2017-08-21 20:53:18,772 - test - INFO - SSH address = lotus-59vm13.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:53:19,478 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config restart
2017-08-21 20:53:19,479 - test - INFO - SSH address = lotus-59vm13.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:53:59,844 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:53:59,845 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:00,249 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:54:00,249 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:00,633 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:54:00,633 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:01,037 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:54:01,037 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:01,422 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:54:01,423 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:01,810 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:54:01,811 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:02,195 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:54:02,195 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:02,598 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:54:02,599 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:02,982 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:54:02,982 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:03,689 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_access_control (tests.integration.shared_storage_configuration.test_anonymous_access_control.TestAnonymousAccessControl) ====="
2017-08-21 20:54:03,690 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:04,596 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:54:04,596 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:54:04,597 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:04,998 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 20:54:04,999 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:54:05,000 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:05,375 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:54:05,375 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:54:05,375 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:05,778 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 20:54:05,778 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:54:05,778 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:06,154 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:54:06,154 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:54:06,154 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:06,552 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 20:54:06,553 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:54:06,553 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:06,937 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:54:06,937 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:54:06,937 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:07,341 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 20:54:07,341 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:54:07,341 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:07,730 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:54:07,731 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:54:07,731 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:08,438 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 20:54:08,438 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:54:08,438 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:09,133 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:54:09,133 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:09,840 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:54:09,840 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:10,532 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:54:10,532 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:11,228 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:54:11,228 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:11,930 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:54:11,930 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:12,626 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:54:12,626 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:54:13,331 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 20:54:40,657 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 20:54:42,278 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 20:54:44,023 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 20:55:59,855 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 20:56:02,016 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 20:56:02,573 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 20:56:02,573 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 20:56:02,573 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 20:56:06,650 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:56:06,650 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:07,010 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:56:07,010 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:07,391 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:56:07,392 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:07,766 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:56:07,767 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:08,152 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:56:08,153 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:08,529 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:56:08,529 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:08,901 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:56:08,901 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:09,306 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:56:09,306 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:09,680 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:56:09,680 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:10,345 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:56:10,345 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:10,719 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:56:10,719 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:11,142 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:56:11,142 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:12,379 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:56:12,380 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:13,601 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:56:13,602 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:13,993 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:56:13,994 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:14,396 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:56:14,396 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:14,836 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:56:14,837 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:15,223 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:56:15,223 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:15,624 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:56:15,624 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:16,121 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:56:16,122 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:17,433 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:56:17,433 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:18,709 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:56:18,709 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:19,085 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:56:19,085 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:19,468 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:56:19,468 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:19,904 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:56:19,905 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:20,282 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:56:20,282 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:20,685 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:56:20,685 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:21,160 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:56:21,161 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:22,317 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:56:22,318 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:23,557 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:56:23,557 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:23,935 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:56:23,935 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:24,308 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:56:24,308 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:24,735 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:56:24,735 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:25,100 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:56:25,101 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:25,503 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:56:25,503 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:25,980 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:56:25,981 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:27,172 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:56:27,173 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:28,431 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:56:28,431 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:28,805 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:56:28,805 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:29,176 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:56:29,177 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:29,596 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:56:29,597 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:29,980 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 20:56:29,981 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:56:29,981 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:30,384 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:56:30,384 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:30,787 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:56:30,787 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:31,185 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:56:31,186 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:31,584 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:56:31,585 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:32,289 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:56:32,289 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:32,667 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:56:32,667 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:33,041 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:56:33,042 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:33,418 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:56:33,419 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:33,821 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:56:33,821 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:34,578 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:56:34,578 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:34,982 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:56:34,982 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:35,366 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:56:35,366 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:35,802 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:56:35,802 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:36,203 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:56:36,204 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:36,910 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:56:36,910 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:37,311 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:56:37,311 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:37,702 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:56:37,702 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:38,103 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:56:38,104 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:38,503 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:56:38,504 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:38,908 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:56:38,908 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:39,307 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:56:39,307 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:39,703 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:56:39,704 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:40,095 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:56:40,096 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:40,848 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:56:40,849 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:56:40,850 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:56:40,853 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:56:40,855 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:56:41,247 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:41,249 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:41,289 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:41,330 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:41,743 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:41,744 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:56:41,745 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:56:41,746 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:56:41,746 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:56:41,748 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:56:42,068 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:42,072 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:42,208 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:42,279 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:42,474 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:56:42,475 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:56:42,476 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:56:42,476 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:56:42,477 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:56:42,852 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:56:42,863 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:56:42,969 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:56:42,982 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:56:42,983 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:56:42,984 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:56:42,983 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:56:42,985 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:56:43,300 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:56:43,309 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:56:43,362 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:56:43,475 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:56:43,476 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:56:44,860 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 20:56:44,861 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:56:45,504 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:56:45,504 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:56:45,505 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:56:45,506 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:56:45,507 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:56:45,812 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:56:45,920 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:56:45,924 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:56:46,001 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:56:46,001 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:56:48,350 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 20:56:48,350 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:56:48,884 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:56:48,885 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:56:48,886 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:56:48,887 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:56:48,889 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:56:49,200 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:56:49,307 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:56:49,309 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:56:49,379 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:56:49,379 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:56:52,859 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 20:56:52,859 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:56:53,489 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:56:53,489 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 20:56:53,754 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 20:56:53,755 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:56:54,058 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:56:54,059 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:56:54,358 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:56:54,359 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:56:54,656 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:56:54,657 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:56:54,980 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:56:54,980 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 20:56:55,505 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 20:56:55,506 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:56:55,842 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:56:55,842 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 20:56:56,525 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 20:56:56,526 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:56:56,867 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:56:56,867 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 20:56:57,413 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 20:56:57,414 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:56:57,723 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:56:57,723 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:56:58,015 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:56:58,015 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:56:58,317 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:56:58,317 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:56:58,317 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:58,701 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:56:58,701 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:59,101 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:56:59,101 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:59,503 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:56:59,503 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:56:59,904 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:56:59,904 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:02,348 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 20:57:05,065 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:05,066 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:05,464 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:05,465 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:05,838 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:05,839 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:06,240 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:06,240 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:06,616 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:06,616 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:07,015 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:07,016 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:07,406 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:07,407 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:07,807 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:07,807 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:08,189 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:08,189 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:08,901 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_login (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:08,901 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:09,844 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:57:09,845 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:09,845 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:10,227 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 20:57:10,227 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:10,228 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:10,616 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:57:10,616 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:10,616 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:11,016 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 20:57:11,016 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:11,017 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:11,391 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:57:11,391 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:11,391 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:11,785 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 20:57:11,785 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:11,786 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:12,162 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:57:12,162 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:12,162 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:12,567 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 20:57:12,567 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:12,567 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:12,944 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 20:57:12,944 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:57:12,945 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:13,652 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 20:57:13,652 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 20:57:13,652 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:14,346 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:57:14,346 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:15,055 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:57:15,055 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:15,753 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:57:15,754 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:16,446 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:57:16,446 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:17,150 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 20:57:17,150 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:17,865 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 20:57:17,865 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:57:18,565 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 20:57:41,898 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 20:57:43,518 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 20:57:45,164 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 20:58:59,476 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 20:59:01,220 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 20:59:01,820 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 20:59:01,820 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 20:59:01,820 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 20:59:05,796 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:59:05,797 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:06,195 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:59:06,196 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:06,586 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:59:06,586 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:06,966 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:59:06,967 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:07,343 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:59:07,344 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:07,745 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:59:07,745 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:08,017 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:59:08,018 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:08,390 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 20:59:08,390 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:08,762 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 20:59:08,763 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:09,458 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:59:09,458 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:09,860 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:59:09,860 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:10,349 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:59:10,350 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:11,648 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:59:11,649 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:12,965 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:59:12,966 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:13,323 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:59:13,323 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:13,711 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:59:13,712 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:14,141 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:59:14,142 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:14,517 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:59:14,518 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:14,882 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:59:14,883 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:15,352 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:59:15,352 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:16,557 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:59:16,557 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:17,848 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:59:17,848 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:18,221 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:59:18,222 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:18,596 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:59:18,597 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:19,024 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:59:19,024 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:19,398 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:59:19,398 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:19,779 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:59:19,779 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:20,256 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:59:20,256 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:21,504 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:59:21,504 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:22,751 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:59:22,751 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:23,122 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:59:23,122 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:23,502 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:59:23,502 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:23,924 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:59:23,925 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:24,308 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 20:59:24,309 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:24,709 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 20:59:24,710 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:25,194 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 20:59:25,195 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:26,493 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 20:59:26,494 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:27,736 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 20:59:27,736 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:28,111 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 20:59:28,111 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:28,485 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 20:59:28,486 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:28,910 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 20:59:28,910 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:29,284 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 20:59:29,284 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:59:29,285 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:29,669 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:59:29,670 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:30,073 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:59:30,074 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:30,461 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:59:30,461 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:30,847 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 20:59:30,848 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:31,586 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:59:31,586 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:31,963 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:59:31,964 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:32,347 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:59:32,348 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:32,747 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:59:32,748 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:33,152 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 20:59:33,152 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:33,858 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:59:33,858 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:34,259 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:59:34,259 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:34,660 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:59:34,660 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:35,033 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:59:35,033 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:35,430 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 20:59:35,430 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:36,131 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:59:36,132 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:36,422 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:59:36,422 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:36,820 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:59:36,820 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:37,222 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 20:59:37,222 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:37,597 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:59:37,598 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:37,989 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:59:37,990 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:38,386 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:59:38,387 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:38,671 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:59:38,671 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:39,045 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 20:59:39,046 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:39,800 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:59:39,802 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:59:39,802 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:59:39,803 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:59:39,805 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 20:59:40,177 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:40,230 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:40,355 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:40,356 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:40,585 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:40,586 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:59:40,588 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:59:40,589 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:59:40,590 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:59:40,592 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 20:59:41,010 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:41,052 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:41,055 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:41,058 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:41,413 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 20:59:41,414 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:59:41,415 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:59:41,414 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:59:41,415 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 20:59:41,747 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:59:41,749 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:59:41,826 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:59:41,935 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 20:59:41,936 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:59:41,937 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:59:41,938 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:59:41,939 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:59:42,213 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:59:42,254 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:59:42,361 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:59:42,436 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 20:59:42,437 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:59:43,877 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 20:59:43,878 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:59:44,459 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:59:44,460 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:59:44,462 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:59:44,463 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:59:44,464 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:59:44,863 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:59:44,900 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:59:44,971 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:59:44,973 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 20:59:44,974 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:59:47,323 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 20:59:47,324 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:59:47,965 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:59:47,966 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:59:47,967 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:59:47,968 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:59:47,971 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:59:48,291 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:59:48,363 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:59:48,395 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:59:48,477 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 20:59:48,477 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 20:59:51,856 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 20:59:51,856 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:59:52,368 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:59:52,368 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 20:59:52,634 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 20:59:52,634 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:59:52,919 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:59:52,919 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:59:53,328 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:59:53,328 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:59:53,735 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:59:53,736 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:59:54,066 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:59:54,066 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 20:59:54,645 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 20:59:54,646 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:59:55,119 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:59:55,119 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 20:59:55,701 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 20:59:55,702 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 20:59:56,037 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 20:59:56,038 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 20:59:56,541 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 20:59:56,542 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 20:59:56,864 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 20:59:56,864 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 20:59:57,153 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 20:59:57,153 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 20:59:57,440 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 20:59:57,440 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:59:57,440 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:57,833 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:59:57,834 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:58,194 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:59:58,195 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:58,588 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:59:58,588 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 20:59:58,988 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 20:59:58,988 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:01,420 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 21:00:04,758 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:04,759 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:05,144 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 21:00:05,144 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:05,519 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:05,520 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:05,920 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 21:00:05,920 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:06,292 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:06,293 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:06,690 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 21:00:06,691 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:07,065 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:07,065 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:07,468 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 21:00:07,468 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:07,857 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:07,857 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:08,562 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_user_management (tests.integration.shared_storage_configuration.test_auth.TestAuthentication) ====="
2017-08-21 21:00:08,562 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:09,498 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:00:09,499 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:09,499 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:09,872 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 21:00:09,872 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:00:09,873 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:10,254 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:00:10,254 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:10,254 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:10,653 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 21:00:10,653 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:00:10,653 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:11,026 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:00:11,027 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:11,027 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:11,430 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 21:00:11,430 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:00:11,430 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:11,805 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:00:11,805 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:11,805 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:12,199 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 21:00:12,199 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:00:12,199 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:12,622 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:00:12,623 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:00:12,623 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:13,328 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 21:00:13,329 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:00:13,329 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:13,973 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:00:13,974 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:14,677 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:00:14,677 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:15,394 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:00:15,395 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:16,095 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:00:16,096 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:16,796 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:00:16,797 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:17,490 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:00:17,491 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:00:18,192 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 21:00:41,513 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 21:00:43,095 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 21:00:44,854 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 21:01:59,112 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 21:02:01,046 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 21:02:01,678 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 21:02:01,679 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 21:02:01,679 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 21:02:05,790 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:02:05,790 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:06,179 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:02:06,179 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:06,551 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:02:06,552 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:06,954 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:02:06,954 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:07,309 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:02:07,309 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:07,680 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:02:07,680 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:08,068 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:02:08,069 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:08,474 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:02:08,475 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:08,854 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:02:08,854 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:09,561 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:02:09,562 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:09,965 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:02:09,965 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:10,448 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:02:10,448 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:11,731 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:02:11,731 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:12,920 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:02:12,921 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:13,289 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:02:13,289 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:13,656 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:02:13,656 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:14,083 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:02:14,083 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:14,459 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:02:14,460 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:14,859 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:02:14,859 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:15,335 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:02:15,335 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:16,596 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:02:16,596 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:17,835 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:02:17,836 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:18,206 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:02:18,206 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:18,581 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:02:18,582 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:19,005 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:02:19,006 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:19,379 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:02:19,380 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:19,773 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:02:19,774 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:20,244 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:02:20,244 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:21,443 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:02:21,443 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:22,630 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:02:22,631 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:23,025 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:02:23,025 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:23,389 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:02:23,389 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:23,810 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:02:23,810 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:24,203 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:02:24,203 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:24,603 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:02:24,603 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:25,089 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:02:25,090 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:26,382 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:02:26,382 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:27,699 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:02:27,699 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:28,075 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:02:28,075 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:28,463 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:02:28,463 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:28,852 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:02:28,853 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:29,237 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 21:02:29,237 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:02:29,238 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:29,643 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:02:29,643 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:30,042 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:02:30,042 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:30,444 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:02:30,445 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:30,847 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:02:30,847 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:31,548 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:02:31,548 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:31,950 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:02:31,950 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:32,349 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:02:32,350 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:32,748 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:02:32,748 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:33,148 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:02:33,148 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:33,850 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:02:33,850 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:34,229 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:02:34,230 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:34,613 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:02:34,613 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:35,014 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:02:35,014 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:35,404 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:02:35,404 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:36,108 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:02:36,108 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:36,505 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:02:36,506 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:36,907 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:02:36,908 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:37,305 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:02:37,305 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:37,682 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:02:37,683 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:38,135 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:02:38,136 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:38,535 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:02:38,536 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:38,912 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:02:38,913 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:39,314 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:02:39,314 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:40,122 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:02:40,123 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:02:40,123 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:02:40,124 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:02:40,126 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:02:40,473 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:40,583 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:40,604 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:40,659 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:40,854 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:40,855 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:02:40,856 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:02:40,855 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:02:40,857 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:02:40,857 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:02:41,261 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:41,281 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:41,333 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:41,434 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:41,658 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:02:41,659 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:02:41,660 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:02:41,660 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:02:41,659 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:02:42,066 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:02:42,169 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:02:42,170 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:02:42,174 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:02:42,175 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:02:42,177 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:02:42,178 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:02:42,179 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:02:42,439 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:02:42,455 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:02:42,565 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:02:42,635 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:02:42,635 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:02:43,975 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 21:02:43,975 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:02:44,496 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:02:44,497 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:02:44,498 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:02:44,499 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:02:44,500 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:02:44,819 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:02:44,863 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:02:44,883 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:02:44,893 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:02:44,893 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:02:47,227 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 21:02:47,227 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:02:47,753 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:02:47,753 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:02:47,754 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:02:47,755 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:02:47,756 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:02:48,169 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:02:48,171 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:02:48,177 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:02:48,204 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:02:48,204 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:02:51,680 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 21:02:51,680 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:02:52,296 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:02:52,296 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 21:02:52,559 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 21:02:52,560 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:02:52,971 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:02:52,971 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:02:53,392 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:02:53,392 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:02:53,790 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:02:53,791 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:02:54,245 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:02:54,245 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 21:02:54,886 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 21:02:54,887 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:02:55,215 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:02:55,215 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 21:02:55,783 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 21:02:55,784 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:02:56,223 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:02:56,223 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 21:02:56,746 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 21:02:56,747 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:02:57,037 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:02:57,037 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:02:57,428 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:02:57,428 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:02:57,730 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:02:57,730 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:02:57,731 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:58,102 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:02:58,102 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:58,477 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:02:58,477 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:58,853 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:02:58,853 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:02:59,228 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:02:59,228 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:03:02,041 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 21:03:02,360 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T04:03:01.474599', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 21:03:10,954 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T04:03:01.474599', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 21:03:11,967 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': False, u'created_at': u'2017-08-22T04:03:10.902259', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 21:03:19,273 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T04:03:10.902259', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 21:03:19,709 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:20,756 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:21,787 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:22,827 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:23,864 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:24,910 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:25,949 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:26,985 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:28,022 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:29,061 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:30,101 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:31,139 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:32,176 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:33,223 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:34,260 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:35,294 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:36,334 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:37,372 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:38,412 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:39,453 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:40,489 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:41,528 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:42,564 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:43,603 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:44,640 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:45,681 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:46,714 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:47,755 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:48,796 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:49,838 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:50,877 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:51,915 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:52,951 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:53,990 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:55,030 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:56,068 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:57,107 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:58,146 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:03:59,192 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:00,229 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:01,270 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:02,309 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:03,350 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:04,392 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:05,434 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:06,472 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:07,511 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:08,542 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:09,579 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:10,616 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:11,657 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:12,696 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:13,735 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:14,774 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:15,816 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:16,847 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:17,885 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:18,923 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:19,960 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:20,998 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:22,037 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:23,073 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:24,110 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:25,150 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:26,190 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:27,228 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:28,257 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:29,294 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:30,322 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:31,356 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:32,403 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:33,443 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:34,474 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:04:35,904 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': False, u'created_at': u'2017-08-22T04:04:34.856131', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 21:15:48,131 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': True, u'created_at': u'2017-08-22T04:04:34.856131', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 21:15:48,510 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:15:48,511 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:49,373 - test - DEBUG - set_state /api/lnet_configuration/1/ lnet_up
2017-08-21 21:15:49,508 - test - WARNING - set_state /api/lnet_configuration/1/ lnet_up - no-op
2017-08-21 21:15:49,508 - test - DEBUG - assertValue state /api/lnet_configuration/1/ lnet_up
2017-08-21 21:15:49,777 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:49,777 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:50,172 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:50,172 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:50,561 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:50,562 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:50,958 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:50,958 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:51,334 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:51,335 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:51,739 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:51,740 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:52,115 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:52,115 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:52,519 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:52,519 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:52,903 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:52,904 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:53,608 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_actions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:53,608 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:54,505 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:15:54,505 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:54,505 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:54,882 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 21:15:54,882 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:54,882 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:55,288 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:15:55,288 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:55,288 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:55,689 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 21:15:55,689 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:55,690 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:56,084 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:15:56,085 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:56,085 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:56,487 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 21:15:56,487 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:56,488 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:56,855 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:15:56,855 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:56,855 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:57,239 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 21:15:57,239 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:57,240 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:57,615 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:15:57,615 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:15:57,615 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:58,317 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 21:15:58,317 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:15:58,318 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:59,005 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:15:59,005 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:15:59,712 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:15:59,712 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:16:00,404 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:16:00,404 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:16:01,089 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:16:01,090 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:16:01,784 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:16:01,784 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:16:02,467 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:16:02,468 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:16:03,164 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 21:16:28,493 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 21:16:30,237 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 21:16:31,919 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 21:17:44,549 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 21:17:46,247 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 21:17:46,787 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 21:17:46,787 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 21:17:46,788 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 21:17:50,879 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:17:50,880 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:51,283 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:17:51,283 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:52,836 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:17:52,836 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:53,232 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:17:53,232 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:53,606 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:17:53,607 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:54,013 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:17:54,013 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:54,389 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:17:54,389 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:54,790 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:17:54,791 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:55,167 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:17:55,168 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:55,874 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:17:55,875 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:56,259 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:17:56,260 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:57,839 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:17:57,839 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:17:59,125 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:17:59,126 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:00,420 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:18:00,420 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:00,820 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:18:00,820 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:01,222 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:18:01,222 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:01,676 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:18:01,676 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:02,053 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:18:02,053 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:02,430 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:18:02,431 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:02,905 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:18:02,906 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:04,204 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:18:04,204 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:05,444 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:18:05,444 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:05,821 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:18:05,821 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:06,195 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:18:06,195 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:06,612 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:18:06,612 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:06,996 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:18:06,996 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:07,393 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:18:07,393 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:07,816 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:18:07,817 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:09,086 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:18:09,086 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:10,379 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:18:10,380 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:10,757 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:18:10,757 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:11,142 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:18:11,142 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:11,567 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:18:11,568 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:11,958 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:18:11,959 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:12,361 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:18:12,361 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:12,836 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:18:12,836 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:14,150 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:18:14,151 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:15,397 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:18:15,398 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:15,781 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:18:15,782 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:16,159 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:18:16,159 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:16,585 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:18:16,585 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:16,965 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 21:18:16,965 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:18:16,966 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:17,370 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:18:17,370 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:17,769 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:18:17,769 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:18,169 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:18:18,169 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:18,572 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:18:18,573 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:19,275 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:18:19,276 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:19,675 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:18:19,676 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:20,056 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:18:20,056 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:20,459 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:18:20,459 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:20,836 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:18:20,836 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:21,535 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:18:21,535 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:21,937 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:18:21,937 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:22,333 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:18:22,333 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:22,734 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:18:22,735 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:23,138 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:18:23,138 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:23,846 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:18:23,846 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:24,247 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:18:24,248 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:24,652 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:18:24,652 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:25,056 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:18:25,057 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:25,442 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:18:25,443 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:25,892 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:18:25,892 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:26,277 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:18:26,277 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:26,680 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:18:26,680 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:27,081 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:18:27,081 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:27,834 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:18:27,836 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:18:27,837 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:18:27,839 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:18:27,841 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:18:28,214 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:28,217 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:28,322 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:28,327 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:28,715 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:28,716 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:18:28,716 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:18:28,717 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:18:28,717 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:18:28,718 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:18:29,144 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:29,147 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:29,151 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:29,189 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:29,586 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:18:29,587 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:18:29,588 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:18:29,589 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:18:29,590 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:18:29,922 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:18:29,988 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:18:29,994 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:18:29,996 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:18:29,997 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:18:29,997 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:18:29,999 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:18:30,000 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:18:30,372 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:18:30,399 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:18:30,403 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:18:30,470 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:18:30,471 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:18:31,833 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 21:18:31,834 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:18:32,470 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:18:32,471 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:18:32,472 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:18:32,473 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:18:32,475 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:18:32,767 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:18:32,921 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:18:32,942 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:18:33,002 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:18:33,003 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:18:35,483 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 21:18:35,484 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:18:36,007 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:18:36,008 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:18:36,009 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:18:36,011 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:18:36,012 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:18:36,424 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:18:36,457 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:18:36,540 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:18:36,541 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:18:36,542 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:18:40,001 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 21:18:40,001 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:18:40,491 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:18:40,491 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 21:18:40,761 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 21:18:40,761 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:18:41,061 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:18:41,061 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:18:41,348 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:18:41,348 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:18:41,658 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:18:41,659 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:18:41,991 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:18:41,991 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 21:18:42,465 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 21:18:42,466 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:18:42,920 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:18:42,920 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 21:18:43,465 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 21:18:43,465 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:18:43,796 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:18:43,796 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 21:18:44,517 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 21:18:44,518 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:18:44,919 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:18:44,919 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:18:45,213 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:18:45,213 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:18:45,498 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:18:45,498 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:18:45,498 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:45,873 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:18:45,874 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:46,271 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:18:46,271 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:46,672 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:18:46,673 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:47,068 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:18:47,068 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:18:50,089 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 21:18:51,112 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T04:18:49.530017', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 21:18:58,080 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T04:18:49.530017', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 21:18:59,235 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': False, u'created_at': u'2017-08-22T04:18:58.149090', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 21:19:06,531 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T04:18:58.149090', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 21:19:06,985 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:08,027 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:09,068 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:10,107 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:11,157 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:12,207 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:13,246 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:14,287 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:15,328 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:16,369 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:17,409 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:18,448 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:19,490 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:20,532 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:21,572 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:22,613 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:23,654 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:24,693 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:25,734 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:26,773 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:27,815 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:28,857 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:29,896 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:30,936 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:31,977 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:33,018 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:34,054 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:35,094 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:36,131 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:37,172 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:38,212 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:39,253 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:40,292 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:41,333 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:42,373 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:43,411 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:44,452 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:45,492 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:46,531 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:47,573 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:48,611 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:49,653 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:50,694 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:51,735 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:52,776 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:53,817 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:54,856 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:55,899 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:56,938 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:57,977 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:19:59,019 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:00,057 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:01,095 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:02,135 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:03,174 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:04,211 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:05,234 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:06,275 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:07,315 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:08,353 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:09,392 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:10,431 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:11,467 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:12,503 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:13,539 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:14,575 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:15,614 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:16,653 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:17,694 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:18,736 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:19,777 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:20,828 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:21,868 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:22,904 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:23,940 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:20:25,373 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': False, u'created_at': u'2017-08-22T04:20:24.328212', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 21:30:51,388 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': True, u'created_at': u'2017-08-22T04:20:24.328212', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 21:30:51,787 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:30:51,787 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:53,232 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:30:53,232 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:53,610 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:30:53,611 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:53,984 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:30:53,984 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:54,388 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:30:54,388 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:54,762 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:30:54,762 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:55,166 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:30:55,167 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:55,540 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:30:55,540 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:55,945 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:30:55,946 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:56,340 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:30:56,340 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:57,053 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_jobs (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:30:57,054 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:57,997 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:30:57,997 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:30:57,997 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:58,347 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 21:30:58,347 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:30:58,348 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:58,720 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:30:58,720 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:30:58,721 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:59,103 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 21:30:59,103 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:30:59,104 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:59,492 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:30:59,492 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:30:59,492 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:30:59,865 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 21:30:59,866 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:30:59,866 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:00,245 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:31:00,246 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:31:00,246 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:00,628 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 21:31:00,628 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:31:00,628 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:01,002 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:31:01,002 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:31:01,002 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:01,758 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 21:31:01,759 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:31:01,759 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:02,451 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:31:02,451 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:03,119 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:31:03,119 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:03,819 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:31:03,819 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:04,509 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:31:04,509 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:05,209 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:31:05,209 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:05,858 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:31:05,858 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:31:06,548 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 21:31:31,805 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 21:31:33,495 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 21:31:35,163 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 21:32:50,894 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 21:32:52,810 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 21:32:53,446 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 21:32:53,446 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 21:32:53,446 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 21:32:57,185 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:32:57,186 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:32:57,591 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:32:57,592 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:32:58,143 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:32:58,143 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:32:58,545 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:32:58,545 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:32:58,921 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:32:58,921 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:32:59,310 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:32:59,310 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:32:59,684 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:32:59,684 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:00,086 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:33:00,087 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:00,361 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:33:00,361 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:01,062 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:33:01,062 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:01,467 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:33:01,467 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:03,061 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:33:03,061 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:04,381 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:33:04,381 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:05,646 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:33:05,646 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:06,037 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:33:06,037 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:06,411 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:33:06,411 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:06,836 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:33:06,836 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:07,210 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:33:07,210 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:07,612 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:33:07,613 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:08,088 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:33:08,089 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:09,330 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:33:09,330 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:10,623 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:33:10,623 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:11,001 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:33:11,001 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:11,393 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:33:11,393 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:11,825 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:33:11,826 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:12,200 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:33:12,200 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:12,603 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:33:12,604 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:13,080 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:33:13,080 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:14,322 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:33:14,322 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:15,585 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:33:15,586 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:15,959 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:33:15,960 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:16,333 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:33:16,334 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:16,756 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:33:16,757 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:17,121 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:33:17,122 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:17,485 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:33:17,486 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:17,962 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:33:17,963 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:19,251 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:33:19,252 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:20,509 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:33:20,510 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:20,884 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:33:20,885 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:21,258 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:33:21,259 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:21,684 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:33:21,684 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:22,061 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 21:33:22,061 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:33:22,061 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:22,467 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:33:22,467 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:22,870 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:33:22,870 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:23,250 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:33:23,250 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:23,627 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:33:23,627 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:24,330 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:33:24,331 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:24,737 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:33:24,737 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:25,140 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:33:25,140 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:25,541 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:33:25,541 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:25,917 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:33:25,917 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:26,623 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:33:26,623 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:27,026 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:33:27,027 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:27,429 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:33:27,429 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:27,818 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:33:27,819 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:28,196 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:33:28,196 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:28,896 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:33:28,897 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:29,297 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:33:29,297 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:29,697 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:33:29,697 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:30,099 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:33:30,099 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:30,501 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:33:30,502 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:30,905 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:33:30,905 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:31,305 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:33:31,306 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:31,703 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:33:31,704 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:32,107 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:33:32,108 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:32,858 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:33:32,859 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:33:32,860 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:33:32,861 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:33:32,861 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:33:33,204 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:33,208 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:33,209 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:33,394 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:33,565 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:33,566 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:33:33,567 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:33:33,567 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:33:33,568 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:33:33,572 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:33:33,904 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:33,973 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:34,021 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:34,036 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:34,537 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:33:34,538 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:33:34,539 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:33:34,540 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:33:34,540 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:33:34,957 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:33:34,960 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:33:34,961 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:33:35,068 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:33:35,069 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:33:35,069 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:33:35,070 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:33:35,071 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:33:35,491 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:33:35,533 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:33:35,609 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:33:35,611 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:33:35,612 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:33:36,998 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 21:33:36,999 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:33:37,666 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:33:37,667 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:33:37,668 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:33:37,668 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:33:37,669 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:33:37,983 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:33:38,039 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:33:38,091 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:33:38,152 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:33:38,153 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:33:40,520 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 21:33:40,520 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:33:41,047 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:33:41,048 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:33:41,049 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:33:41,050 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:33:41,052 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:33:41,348 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:33:41,368 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:33:41,368 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:33:41,533 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:33:41,534 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:33:45,002 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 21:33:45,002 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:33:45,629 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:33:45,629 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 21:33:46,026 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 21:33:46,027 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:33:46,319 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:33:46,319 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:33:46,632 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:33:46,632 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:33:46,924 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:33:46,925 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:33:47,255 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:33:47,256 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 21:33:47,803 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 21:33:47,804 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:33:48,278 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:33:48,278 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 21:33:48,884 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 21:33:48,885 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:33:49,226 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:33:49,227 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 21:33:49,763 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 21:33:49,764 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:33:50,200 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:33:50,200 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:33:50,504 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:33:50,505 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:33:50,790 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:33:50,790 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:33:50,791 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:51,158 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:33:51,158 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:51,560 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:33:51,560 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:51,927 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:33:51,928 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:52,328 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:33:52,329 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:33:55,236 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 21:33:56,218 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T04:33:55.298038', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 21:34:04,229 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T04:33:55.298038', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 21:34:05,336 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': False, u'created_at': u'2017-08-22T04:34:04.224659', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 21:34:13,676 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T04:34:04.224659', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 21:34:13,995 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:15,043 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:16,083 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:17,133 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:18,172 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:19,222 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:20,260 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:21,298 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:22,335 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:23,375 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:24,422 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:25,463 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:26,503 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:27,547 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:28,587 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:29,640 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:30,682 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:31,722 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:32,763 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:33,800 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:34,842 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:35,882 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:36,921 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:37,960 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:39,001 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:40,040 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:41,080 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:42,118 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:43,154 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:44,194 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:45,234 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:46,273 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:47,316 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:48,358 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:49,399 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:50,440 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:51,479 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:52,520 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:53,558 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:54,602 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:55,644 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:56,687 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:57,731 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:58,771 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:34:59,813 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:35:01,283 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': False, u'created_at': u'2017-08-22T04:35:00.199088', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 21:45:46,250 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': True, u'created_at': u'2017-08-22T04:35:00.199088', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 21:45:46,705 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:45:46,705 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:47,754 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:47,754 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:48,155 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:45:48,155 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:48,556 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:48,556 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:48,960 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:45:48,960 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:49,349 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:49,349 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:49,751 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:45:49,751 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:50,132 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:50,132 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:50,502 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:45:50,502 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:50,877 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:50,877 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:51,576 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_available_transitions (tests.integration.shared_storage_configuration.test_available_jobs_and_trans.TestAvailable) ====="
2017-08-21 21:45:51,577 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:52,503 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:45:52,504 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:52,504 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:52,896 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 21:45:52,896 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:45:52,897 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:53,274 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:45:53,274 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:53,274 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:53,677 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 21:45:53,677 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:45:53,677 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:54,060 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:45:54,060 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:54,061 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:54,464 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 21:45:54,465 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:45:54,466 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:54,850 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:45:54,850 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:54,851 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:55,251 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 21:45:55,251 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:45:55,251 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:55,631 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:45:55,632 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:45:55,632 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:56,334 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 21:45:56,335 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:45:56,335 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:57,020 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:45:57,021 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:57,720 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:45:57,720 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:58,377 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:45:58,377 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:59,069 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:45:59,070 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:45:59,771 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:45:59,771 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:46:00,471 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:46:00,471 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:46:01,170 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 21:46:30,582 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 21:46:32,175 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 21:46:33,756 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 21:47:47,418 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 21:47:49,361 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 21:47:49,878 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 21:47:49,878 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 21:47:49,878 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 21:47:53,672 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:47:53,673 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:54,062 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:47:54,062 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:54,532 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:47:54,532 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:54,906 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:47:54,907 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:55,280 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:47:55,280 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:55,655 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:47:55,655 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:56,027 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:47:56,028 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:56,403 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:47:56,404 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:56,775 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:47:56,776 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:57,473 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:47:57,474 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:57,859 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:47:57,859 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:47:59,439 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:47:59,440 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:00,679 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:48:00,680 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:01,945 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:48:01,945 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:02,337 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:48:02,338 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:02,732 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:48:02,733 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:03,185 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:48:03,185 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:03,579 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:48:03,579 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:03,979 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:48:03,979 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:04,453 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:48:04,453 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:05,710 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:48:05,710 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:06,951 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:48:06,952 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:07,325 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:48:07,325 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:07,701 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:48:07,701 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:08,126 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:48:08,126 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:08,499 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:48:08,499 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:08,877 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:48:08,877 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:09,353 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:48:09,353 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:10,603 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:48:10,604 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:11,896 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:48:11,896 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:12,265 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:48:12,265 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:12,646 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:48:12,646 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:13,070 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:48:13,071 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:13,432 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:48:13,433 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:13,808 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:48:13,808 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:14,298 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:48:14,298 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:15,602 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:48:15,603 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:16,952 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:48:16,953 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:17,339 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:48:17,339 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:17,710 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:48:17,711 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:18,135 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:48:18,135 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:18,522 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 21:48:18,522 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:48:18,523 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:18,920 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:48:18,920 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:19,319 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:48:19,319 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:19,694 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:48:19,694 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:20,070 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:48:20,070 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:20,762 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:48:20,762 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:21,153 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:48:21,154 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:21,523 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:48:21,524 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:21,893 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:48:21,894 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:22,266 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:48:22,267 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:22,958 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:48:22,958 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:23,350 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:48:23,350 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:23,726 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:48:23,727 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:24,103 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:48:24,104 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:24,476 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:48:24,477 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:25,173 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:48:25,173 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:25,568 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:48:25,569 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:25,954 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:48:25,955 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:26,330 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:48:26,330 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:26,700 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:48:26,700 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:27,089 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:48:27,089 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:27,349 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:48:27,349 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:27,716 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:48:27,716 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:28,108 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:48:28,108 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:28,848 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:48:28,849 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:48:28,851 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:48:28,852 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:48:28,855 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:48:29,313 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:29,368 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:29,379 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:29,394 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:29,657 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:29,658 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:48:29,658 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:48:29,659 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:48:29,661 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:48:29,662 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:48:30,000 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:30,064 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:30,190 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:30,251 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:30,670 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:48:30,671 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:48:30,671 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:48:30,671 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:48:30,672 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:48:30,984 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:48:31,056 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:48:31,092 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:48:31,170 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:48:31,172 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:48:31,172 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:48:31,173 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:48:31,174 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:48:31,494 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:48:31,567 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:48:31,573 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:48:31,610 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:48:31,610 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:48:33,068 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 21:48:33,069 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:48:33,541 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:48:33,542 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:48:33,543 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:48:33,544 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:48:33,545 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:48:33,906 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:48:33,926 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:48:33,992 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:48:33,996 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:48:33,997 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:48:36,355 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 21:48:36,355 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:48:36,979 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:48:36,980 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:48:36,981 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:48:36,982 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:48:36,982 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:48:37,346 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:48:37,354 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:48:37,389 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:48:37,454 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:48:37,455 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:48:40,799 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 21:48:40,799 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:48:41,272 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:48:41,272 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 21:48:41,544 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 21:48:41,544 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:48:41,951 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:48:41,952 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:48:42,235 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:48:42,235 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:48:42,507 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:48:42,508 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:48:42,826 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:48:42,826 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 21:48:43,449 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 21:48:43,450 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:48:43,886 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:48:43,886 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 21:48:44,411 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 21:48:44,412 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:48:44,851 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:48:44,851 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 21:48:45,472 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 21:48:45,473 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:48:45,774 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:48:45,774 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:48:46,073 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:48:46,074 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:48:46,465 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:48:46,465 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:48:46,466 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:46,854 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:48:46,854 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:47,222 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:48:47,222 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:47,611 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:48:47,612 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:47,983 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:48:47,984 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:50,832 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 21:48:50,929 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-diagnostics -v -v -v
2017-08-21 21:48:50,930 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:57,075 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:48:57,075 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:57,444 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:48:57,444 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:57,830 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:48:57,831 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:58,232 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:48:58,233 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:58,606 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:48:58,606 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:58,982 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:48:58,983 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:59,356 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:48:59,356 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:48:59,854 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:48:59,854 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:00,248 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:49:00,248 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:00,912 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_with_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:49:00,913 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:02,497 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:49:02,498 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:49:02,498 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:02,866 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 21:49:02,867 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:49:02,867 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:03,227 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:49:03,227 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:49:03,227 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:03,613 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 21:49:03,613 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:49:03,614 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:03,988 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:49:03,988 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:49:03,989 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:04,342 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 21:49:04,342 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:49:04,343 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:04,736 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:49:04,737 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:49:04,737 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:05,119 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 21:49:05,119 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:49:05,120 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:05,505 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:49:05,505 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:49:05,505 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:06,207 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 21:49:06,208 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:49:06,208 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:06,901 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:49:06,902 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:07,597 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:49:07,597 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:08,157 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:49:08,158 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:08,716 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:49:08,716 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:09,376 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:49:09,377 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:10,068 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:49:10,068 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:49:10,758 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 21:49:36,030 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 21:49:37,773 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 21:49:39,576 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 21:50:54,291 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 21:50:56,176 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 21:50:56,830 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 21:50:56,831 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 21:50:56,831 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 21:51:00,753 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:51:00,754 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:01,204 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:51:01,204 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:01,579 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:51:01,580 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:01,957 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:51:01,957 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:02,330 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:51:02,330 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:02,735 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:51:02,735 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:03,111 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:51:03,111 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:03,513 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:51:03,513 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:03,896 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:51:03,896 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:04,607 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:51:04,608 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:04,994 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:51:04,994 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:05,469 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:51:05,470 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:06,813 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:51:06,813 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:08,127 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:51:08,128 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:08,501 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:51:08,501 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:08,886 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:51:08,886 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:09,310 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:51:09,311 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:09,695 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:51:09,695 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:10,098 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:51:10,098 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:10,574 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:51:10,575 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:11,818 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:51:11,818 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:13,069 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:51:13,069 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:13,459 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:51:13,459 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:13,833 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:51:13,833 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:14,259 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:51:14,259 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:14,635 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:51:14,635 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:15,035 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:51:15,036 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:15,512 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:51:15,512 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:16,811 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:51:16,811 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:18,129 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:51:18,129 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:18,508 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:51:18,509 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:18,884 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:51:18,885 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:19,319 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:51:19,319 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:19,691 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:51:19,692 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:20,068 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:51:20,069 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:20,513 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:51:20,513 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:21,709 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:51:21,709 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:23,010 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:51:23,010 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:23,401 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:51:23,401 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:23,777 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:51:23,777 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:24,203 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:51:24,203 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:24,591 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 21:51:24,591 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:51:24,591 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:24,993 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:51:24,994 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:25,397 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:51:25,398 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:25,799 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:51:25,799 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:26,200 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:51:26,200 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:26,904 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:51:26,905 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:27,289 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:51:27,289 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:27,666 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:51:27,667 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:28,043 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:51:28,043 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:28,418 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:51:28,419 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:29,124 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:51:29,124 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:29,510 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:51:29,511 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:29,884 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:51:29,885 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:30,259 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:51:30,259 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:30,634 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:51:30,634 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:31,335 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:51:31,335 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:31,721 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:51:31,722 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:32,124 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:51:32,124 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:32,525 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:51:32,526 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:32,922 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:51:32,922 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:33,298 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:51:33,299 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:33,701 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:51:33,701 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:34,085 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:51:34,086 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:34,486 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:51:34,487 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:35,237 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:51:35,238 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:51:35,239 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:51:35,239 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:51:35,242 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:51:35,619 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:35,751 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:35,810 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:35,812 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:36,098 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:36,099 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:51:36,101 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:51:36,100 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:51:36,102 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:51:36,103 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:51:36,549 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:36,564 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:36,568 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:36,571 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:36,983 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:51:36,984 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:51:36,985 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:51:36,986 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:51:36,986 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:51:37,380 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:51:37,392 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:51:37,429 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:51:37,504 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:51:37,505 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:51:37,507 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:51:37,508 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:51:37,510 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:51:37,883 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:51:37,931 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:51:37,996 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:51:38,000 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:51:38,001 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:51:39,371 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 21:51:39,371 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:51:39,879 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:51:39,880 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:51:39,881 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:51:39,880 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:51:39,882 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:51:40,309 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:51:40,315 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:51:40,389 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:51:40,390 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:51:40,391 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:51:42,857 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 21:51:42,857 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:51:43,332 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:51:43,333 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:51:43,334 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:51:43,335 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:51:43,336 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:51:43,574 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:51:43,716 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:51:43,751 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:51:43,829 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:51:43,829 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:51:47,177 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 21:51:47,177 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:51:47,782 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:51:47,783 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 21:51:48,049 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 21:51:48,049 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:51:48,348 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:51:48,349 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:51:48,651 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:51:48,651 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:51:48,922 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:51:48,923 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:51:49,239 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:51:49,239 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 21:51:49,748 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 21:51:49,749 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:51:50,216 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:51:50,216 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 21:51:50,824 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 21:51:50,825 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:51:51,149 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:51:51,149 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 21:51:51,639 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 21:51:51,640 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:51:51,913 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:51:51,913 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:51:52,293 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:51:52,293 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:51:52,704 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:51:52,705 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:51:52,705 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:53,096 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:51:53,096 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:53,469 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:51:53,470 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:53,869 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:51:53,870 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:54,245 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:51:54,245 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:51:56,392 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 21:51:56,506 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: chroma-diagnostics 
2017-08-21 21:51:56,507 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:10,610 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:10,611 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:11,004 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:54:11,004 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:11,378 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:11,378 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:11,752 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:54:11,752 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:12,138 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:12,138 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:12,506 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:54:12,507 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:12,880 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:12,880 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:13,249 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:54:13,250 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:13,623 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:13,624 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:14,342 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_diagnostics_without_sosreport (tests.integration.shared_storage_configuration.test_chroma_diagnotics.TestChromaDiagnostics) ====="
2017-08-21 21:54:14,342 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:15,948 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:54:15,948 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:15,950 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:16,351 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 21:54:16,351 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 21:54:16,351 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:16,740 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:54:16,740 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:16,741 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:17,141 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 21:54:17,141 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 21:54:17,142 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:17,528 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:54:17,529 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:17,529 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:17,953 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 21:54:17,953 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 21:54:17,953 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:18,329 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:54:18,329 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:18,330 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:18,704 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 21:54:18,704 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 21:54:18,704 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:19,093 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:54:19,094 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:19,094 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:19,798 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 21:54:19,798 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 21:54:19,799 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:20,498 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 21:54:20,498 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:54:20,498 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:21,195 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 21:54:21,196 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 21:54:21,196 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:21,856 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:54:21,856 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:22,549 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:54:22,549 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:23,206 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:54:23,206 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:23,905 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:54:23,905 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:24,606 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:54:24,606 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:25,210 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:54:25,210 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:25,870 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:54:25,870 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:26,560 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 21:54:26,561 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:27,216 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 21:54:27,217 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:54:27,917 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 21:54:51,181 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 21:54:52,757 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 21:54:54,488 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 21:56:08,478 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 21:56:10,515 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 21:56:11,086 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 21:56:11,087 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 21:56:11,087 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 21:56:14,987 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:56:14,987 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:15,386 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:56:15,386 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:15,764 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:56:15,764 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:16,137 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:56:16,138 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:16,510 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:56:16,510 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:16,890 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:56:16,890 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:17,139 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:56:17,139 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:17,542 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 21:56:17,543 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:17,926 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:56:17,926 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:18,628 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 21:56:18,628 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:19,336 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:56:19,336 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:19,726 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:56:19,726 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:20,212 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:56:20,212 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:21,506 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:56:21,506 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:22,742 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:56:22,743 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:23,137 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:56:23,137 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:23,530 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:56:23,530 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:23,980 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:56:23,981 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:24,369 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:56:24,369 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:24,758 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:56:24,758 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:25,232 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:56:25,232 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:26,472 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:56:26,473 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:27,712 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:56:27,712 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:28,089 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:56:28,089 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:28,461 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:56:28,461 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:28,900 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:56:28,901 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:29,275 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:56:29,275 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:29,648 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:56:29,648 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:30,141 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:56:30,141 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:31,444 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:56:31,444 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:32,689 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:56:32,689 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:33,078 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:56:33,078 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:33,449 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:56:33,450 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:33,874 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:56:33,874 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:34,255 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 21:56:34,255 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:34,653 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 21:56:34,653 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:35,096 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 21:56:35,096 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:36,321 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 21:56:36,321 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:37,576 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 21:56:37,577 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:37,951 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 21:56:37,951 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:38,335 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 21:56:38,336 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:38,764 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 21:56:38,764 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:39,148 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 21:56:39,149 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 21:56:39,149 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:56:39,149 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:39,533 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:56:39,533 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:39,919 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:56:39,919 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:40,295 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:56:40,296 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:40,695 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:56:40,695 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:41,396 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 21:56:41,396 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:42,092 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:56:42,092 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:42,493 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:56:42,494 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:42,892 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:56:42,893 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:43,267 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:56:43,267 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:43,641 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:56:43,642 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:44,301 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 21:56:44,301 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:44,948 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:56:44,948 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:45,338 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:56:45,338 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:45,703 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:56:45,703 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:46,086 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:56:46,086 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:46,462 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:56:46,462 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:47,119 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 21:56:47,120 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:47,777 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:56:47,778 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:48,174 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:56:48,175 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:48,562 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:56:48,563 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:48,938 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 21:56:48,938 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:49,316 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:56:49,316 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:49,687 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:56:49,687 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:50,061 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:56:50,061 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:50,436 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:56:50,436 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:50,811 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:56:50,811 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:51,357 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 21:56:51,357 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:56:52,023 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:56:52,023 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:56:52,024 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:56:52,026 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:56:52,028 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:56:52,030 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 21:56:52,469 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:52,479 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:52,483 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:52,562 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:52,844 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:52,844 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:52,845 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:56:52,847 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:56:52,848 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:56:52,849 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:56:52,851 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:56:52,855 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 21:56:53,195 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:53,221 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:53,367 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:53,368 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:53,736 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:53,737 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 21:56:53,738 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:56:53,739 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:56:53,740 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:56:53,740 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 21:56:54,099 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:56:54,103 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:56:54,109 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:56:54,216 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 21:56:54,216 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:56:54,218 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:56:54,217 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:56:54,219 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:56:54,511 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:56:54,513 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:56:54,594 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:56:54,708 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 21:56:54,708 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:56:56,183 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 21:56:56,184 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:56:56,671 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:56:56,671 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:56:56,673 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:56:56,673 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:56:56,674 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:56:56,973 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:56:57,064 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:56:57,141 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:56:57,196 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 21:56:57,197 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:56:59,563 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 21:56:59,563 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:57:00,180 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:57:00,181 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:57:00,182 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:57:00,183 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:57:00,184 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:57:00,472 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:57:00,500 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:57:00,562 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:57:00,667 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 21:57:00,667 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 21:57:04,033 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 21:57:04,033 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:57:04,526 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:57:04,527 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 21:57:04,810 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-21 21:57:04,811 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:57:05,095 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:57:05,095 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:57:05,392 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:57:05,392 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:57:05,693 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:57:05,694 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:57:06,051 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:57:06,051 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 21:57:06,743 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 21:57:06,744 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:57:07,099 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:57:07,099 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 21:57:07,588 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 21:57:07,589 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 21:57:08,039 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 21:57:08,040 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 21:57:08,531 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 21:57:08,532 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 21:57:08,825 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 21:57:08,826 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 21:57:09,246 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 21:57:09,247 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 21:57:09,532 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 21:57:09,532 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:57:09,532 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:57:09,928 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:57:09,928 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:57:10,331 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:57:10,332 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:57:10,735 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:57:10,736 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:57:11,126 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:57:11,127 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:57:11,806 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 21:57:11,806 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 21:57:13,996 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 21:57:15,681 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T04:57:14.037432', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 21:57:22,692 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T04:57:14.037432', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 21:57:22,776 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T04:57:14.144250', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 21:57:22,808 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T04:57:14.144250', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 21:57:22,882 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T04:57:14.191014', u'message': u'Validation host configuration lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 21:57:22,912 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T04:57:14.191014', u'message': u'Validation host configuration lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 21:57:22,981 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T04:57:14.252873', u'message': u'Validation host configuration lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 21:57:23,008 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T04:57:14.252873', u'message': u'Validation host configuration lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 21:57:26,408 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/5/'], u'complete': False, u'created_at': u'2017-08-22T04:57:22.943166', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 21:57:34,725 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/5/'], u'complete': True, u'created_at': u'2017-08-22T04:57:22.943166', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 21:57:34,757 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/6/'], u'complete': True, u'created_at': u'2017-08-22T04:57:23.650685', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u''}
2017-08-21 21:57:34,788 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/6/'], u'complete': True, u'created_at': u'2017-08-22T04:57:23.650685', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u''}
2017-08-21 21:57:34,815 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/7/'], u'complete': True, u'created_at': u'2017-08-22T04:57:24.347337', u'message': u'Deploying agent to host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u''}
2017-08-21 21:57:34,841 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/7/'], u'complete': True, u'created_at': u'2017-08-22T04:57:24.347337', u'message': u'Deploying agent to host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u''}
2017-08-21 21:57:34,869 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/8/'], u'complete': True, u'created_at': u'2017-08-22T04:57:24.939906', u'message': u'Deploying agent to host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-21 21:57:34,896 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/8/'], u'complete': True, u'created_at': u'2017-08-22T04:57:24.939906', u'message': u'Deploying agent to host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-21 21:57:35,866 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:36,951 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:38,012 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:39,076 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:40,138 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:41,199 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:42,269 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:43,332 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:44,398 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:45,465 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:46,531 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:47,596 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:48,659 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:49,721 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:50,786 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:51,853 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:52,919 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:53,981 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:55,050 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:56,114 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:57,180 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:58,241 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:57:59,300 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:00,358 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:01,423 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:02,479 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:03,547 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:04,604 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:05,672 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:06,739 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:07,803 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:08,868 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:09,933 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:10,986 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:12,042 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:13,115 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:14,183 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:15,261 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:16,339 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:17,401 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:18,455 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:19,516 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:20,574 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:21,632 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:22,702 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:23,769 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:24,817 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:25,868 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:26,919 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:27,987 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:29,031 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:30,089 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:31,150 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:32,200 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:33,257 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:34,304 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:35,372 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:36,446 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:37,514 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:38,582 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:39,636 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:40,711 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:41,779 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:42,846 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:43,909 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:44,976 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:46,046 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:47,117 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:48,186 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:49,254 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 21:58:51,796 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/'], u'complete': False, u'created_at': u'2017-08-22T04:58:50.055203', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u''}
2017-08-21 22:09:02,351 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/'], u'complete': True, u'created_at': u'2017-08-22T04:58:50.055203', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:09:02,437 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/', u'/api/job/28/', u'/api/job/29/', u'/api/job/30/'], u'complete': True, u'created_at': u'2017-08-22T04:58:50.322792', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:09:02,523 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/', u'/api/job/28/', u'/api/job/29/', u'/api/job/30/'], u'complete': True, u'created_at': u'2017-08-22T04:58:50.322792', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:09:02,603 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/'], u'complete': True, u'created_at': u'2017-08-22T04:58:50.543109', u'message': u'Setup managed host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:09:02,670 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/'], u'complete': True, u'created_at': u'2017-08-22T04:58:50.543109', u'message': u'Setup managed host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:09:02,736 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/', u'/api/job/47/', u'/api/job/48/', u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/'], u'complete': True, u'created_at': u'2017-08-22T04:58:50.757667', u'message': u'Setup managed host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:09:02,801 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/', u'/api/job/47/', u'/api/job/48/', u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/'], u'complete': True, u'created_at': u'2017-08-22T04:58:50.757667', u'message': u'Setup managed host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:09:04,000 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:09:04,000 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:04,402 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:09:04,402 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:04,805 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:09:04,806 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:05,199 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:09:05,200 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:06,743 - test - INFO - Configuring power control on [u'lotus-59vm15.lotus.hpdd.lab.intel.com', u'lotus-59vm16.lotus.hpdd.lab.intel.com', u'lotus-59vm17.lotus.hpdd.lab.intel.com', u'lotus-59vm18.lotus.hpdd.lab.intel.com']
2017-08-21 22:09:07,182 - test - DEBUG - Created /api/power_control_type/14/
2017-08-21 22:09:07,279 - test - DEBUG - Created /api/power_control_device/1/
2017-08-21 22:09:08,185 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/1/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm15', u'id': 1, u'resource_uri': u'/api/power_control_device_outlet/1/'}
2017-08-21 22:09:09,014 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/2/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm16', u'id': 2, u'resource_uri': u'/api/power_control_device_outlet/2/'}
2017-08-21 22:09:09,920 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/3/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm17', u'id': 3, u'resource_uri': u'/api/power_control_device_outlet/3/'}
2017-08-21 22:09:11,066 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/4/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm18', u'id': 4, u'resource_uri': u'/api/power_control_device_outlet/4/'}
2017-08-21 22:09:11,148 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm19', u'id': 5, u'resource_uri': u'/api/power_control_device_outlet/5/'}
2017-08-21 22:09:11,148 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:09:11,149 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:13,797 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: []
2017-08-21 22:09:13,798 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:09:13,798 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:16,382 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: []
2017-08-21 22:09:16,382 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:09:16,382 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:19,042 - test - DEBUG - Fencible nodes on lotus-59vm17.lotus.hpdd.lab.intel.com are: ['lotus-59vm17.lotus.hpdd.lab.intel.com']
2017-08-21 22:09:19,043 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:09:19,043 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:21,643 - test - DEBUG - Fencible nodes on lotus-59vm18.lotus.hpdd.lab.intel.com are: ['lotus-59vm17.lotus.hpdd.lab.intel.com', 'lotus-59vm18.lotus.hpdd.lab.intel.com']
2017-08-21 22:09:22,645 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:09:22,645 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:25,255 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 22:09:25,255 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:09:25,255 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:27,842 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 22:09:27,843 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:09:27,843 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:30,432 - test - DEBUG - Fencible nodes on lotus-59vm17.lotus.hpdd.lab.intel.com are: ['lotus-59vm17.lotus.hpdd.lab.intel.com', 'lotus-59vm18.lotus.hpdd.lab.intel.com']
2017-08-21 22:09:30,432 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:09:30,433 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:09:33,036 - test - DEBUG - Fencible nodes on lotus-59vm18.lotus.hpdd.lab.intel.com are: ['lotus-59vm17.lotus.hpdd.lab.intel.com', 'lotus-59vm18.lotus.hpdd.lab.intel.com']
2017-08-21 22:09:34,666 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/20/', u'volume_nodes': [{u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 4, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/6/', u'id': 6, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/11/', u'id': 11, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 3, u'resource_uri': u'/api/volume_node/16/', u'id': 16, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/48/', u'volume_nodes': [{u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 4, u'resource_uri': u'/api/volume_node/5/', u'id': 5, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/10/', u'id': 10, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/15/', u'id': 15, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 3, u'resource_uri': u'/api/volume_node/20/', u'id': 20, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/44/', u'volume_nodes': [{u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 4, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/13/', u'id': 13, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 3, u'resource_uri': u'/api/volume_node/18/', u'id': 18, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/23/', u'volume_nodes': [{u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 4, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/7/', u'id': 7, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/12/', u'id': 12, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 3, u'resource_uri': u'/api/volume_node/17/', u'id': 17, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/46/', u'volume_nodes': [{u'use': True, u'volume_id': 5, u'primary': True, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 4, u'resource_uri': u'/api/volume_node/4/', u'id': 4, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/14/', u'id': 14, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 3, u'resource_uri': u'/api/volume_node/19/', u'id': 19, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/5/', u'id': 5, u'size': u'10672993730'}]'
2017-08-21 22:09:34,667 - test - DEBUG -         self.wait_until_true(lambda: len(self.get_shared_volumes(required_hosts)) >= expected_volumes)
 evaluated to True
2017-08-21 22:09:36,351 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/20/', u'volume_nodes': [{u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 4, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/6/', u'id': 6, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/11/', u'id': 11, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 3, u'resource_uri': u'/api/volume_node/16/', u'id': 16, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/48/', u'volume_nodes': [{u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 4, u'resource_uri': u'/api/volume_node/5/', u'id': 5, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/10/', u'id': 10, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/15/', u'id': 15, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 3, u'resource_uri': u'/api/volume_node/20/', u'id': 20, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/44/', u'volume_nodes': [{u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 4, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/13/', u'id': 13, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 3, u'resource_uri': u'/api/volume_node/18/', u'id': 18, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/23/', u'volume_nodes': [{u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 4, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/7/', u'id': 7, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/12/', u'id': 12, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 3, u'resource_uri': u'/api/volume_node/17/', u'id': 17, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/46/', u'volume_nodes': [{u'use': True, u'volume_id': 5, u'primary': True, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 4, u'resource_uri': u'/api/volume_node/4/', u'id': 4, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/14/', u'id': 14, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 3, u'resource_uri': u'/api/volume_node/19/', u'id': 19, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/5/', u'id': 5, u'size': u'10672993730'}]'
2017-08-21 22:09:38,041 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/57/', u'/api/job/58/', u'/api/job/59/', u'/api/job/60/', u'/api/job/61/', u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/', u'/api/job/70/', u'/api/job/71/', u'/api/job/72/', u'/api/job/73/'], u'complete': False, u'created_at': u'2017-08-22T05:09:36.316478', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/17/', u'id': 17, u'logs': u'\n\n\n'}
2017-08-21 22:13:21,798 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/57/', u'/api/job/58/', u'/api/job/59/', u'/api/job/60/', u'/api/job/61/', u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/', u'/api/job/70/', u'/api/job/71/', u'/api/job/72/', u'/api/job/73/'], u'complete': True, u'created_at': u'2017-08-22T05:09:36.316478', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/17/', u'id': 17, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:13:21,799 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 22:13:21,799 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:22,183 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 22:13:22,183 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:22,567 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 22:13:22,567 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:22,977 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 22:13:22,977 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:28,468 - test - DEBUG - MGS: should be running on lotus-59vm15.lotus.hpdd.lab.intel.com (actual: lotus-59vm15.lotus.hpdd.lab.intel.com)
2017-08-21 22:13:28,469 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: crm_resource -r MGS_5735d2 -W
2017-08-21 22:13:28,469 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:28,856 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-MGS_5735d2 -N lotus-59vm15.lotus.hpdd.lab.intel.com -G -d 0
2017-08-21 22:13:28,856 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:29,237 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-21 22:13:29,238 - test - DEBUG - testfs-MDT0000: should be running on lotus-59vm16.lotus.hpdd.lab.intel.com (actual: lotus-59vm16.lotus.hpdd.lab.intel.com)
2017-08-21 22:13:29,238 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: crm_resource -r testfs-MDT0000_d0f318 -W
2017-08-21 22:13:29,238 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:29,690 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-testfs-MDT0000_d0f318 -N lotus-59vm16.lotus.hpdd.lab.intel.com -G -d 0
2017-08-21 22:13:29,691 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:30,061 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-21 22:13:30,061 - test - DEBUG - testfs-OST0000: should be running on lotus-59vm17.lotus.hpdd.lab.intel.com (actual: lotus-59vm17.lotus.hpdd.lab.intel.com)
2017-08-21 22:13:30,062 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: crm_resource -r testfs-OST0000_bbf2d2 -W
2017-08-21 22:13:30,062 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:30,443 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-testfs-OST0000_bbf2d2 -N lotus-59vm17.lotus.hpdd.lab.intel.com -G -d 0
2017-08-21 22:13:30,443 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:30,848 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-21 22:13:30,848 - test - DEBUG - testfs-OST0001: should be running on lotus-59vm18.lotus.hpdd.lab.intel.com (actual: lotus-59vm18.lotus.hpdd.lab.intel.com)
2017-08-21 22:13:30,848 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: crm_resource -r testfs-OST0001_a76318 -W
2017-08-21 22:13:30,849 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:31,249 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-testfs-OST0001_a76318 -N lotus-59vm18.lotus.hpdd.lab.intel.com -G -d 0
2017-08-21 22:13:31,250 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:13:31,618 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-21 22:13:31,943 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/78/'], u'complete': False, u'created_at': u'2017-08-22T05:13:31.138601', u'message': u'Validation host configuration lotus-59vm19.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/22/', u'id': 22, u'logs': u''}
2017-08-21 22:13:38,201 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/78/'], u'complete': True, u'created_at': u'2017-08-22T05:13:31.138601', u'message': u'Validation host configuration lotus-59vm19.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/22/', u'id': 22, u'logs': u''}
2017-08-21 22:13:39,449 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/79/'], u'complete': False, u'created_at': u'2017-08-22T05:13:38.363265', u'message': u'Deploying agent to host lotus-59vm19.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/23/', u'id': 23, u'logs': u''}
2017-08-21 22:14:07,519 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/79/'], u'complete': True, u'created_at': u'2017-08-22T05:13:38.363265', u'message': u'Deploying agent to host lotus-59vm19.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/23/', u'id': 23, u'logs': u''}
2017-08-21 22:14:09,492 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/80/', u'/api/job/81/', u'/api/job/82/', u'/api/job/83/', u'/api/job/84/', u'/api/job/85/', u'/api/job/86/'], u'complete': False, u'created_at': u'2017-08-22T05:14:08.511678', u'message': u'Setup worker host lotus-59vm19.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/24/', u'id': 24, u'logs': u''}
2017-08-21 22:26:50,349 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/80/', u'/api/job/81/', u'/api/job/82/', u'/api/job/83/', u'/api/job/84/', u'/api/job/85/', u'/api/job/86/'], u'complete': True, u'created_at': u'2017-08-22T05:14:08.511678', u'message': u'Setup worker host lotus-59vm19.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/24/', u'id': 24, u'logs': u'\n\nReboot of lotus-59vm19.lotus.hpdd.lab.intel.com required to switch from running kernel kernel-3.10.0-693.el7.x86_64 to required kernel-3.10.0-514.26.2.el7.x86_64\n\n\n\n\n\n\n'}
2017-08-21 22:26:52,108 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:26:52,109 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:26:55,157 - test - DEBUG -         self.wait_until_true(lambda: self._get_mount_job(mount_job_name))
 evaluated to {u'verb': u'Mount Filesystem(s)', u'confirmation': u'Mount all associated Lustre filesystems on this worker.', u'class_name': u'MountLustreFilesystemsJob', u'display_order': 120, u'args': {u'host_id': 5}, u'display_group': 3, u'long_description': u'Mount all associated Lustre filesystems on this worker.'}
2017-08-21 22:26:55,615 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/87/'], u'complete': False, u'created_at': u'2017-08-22T05:26:54.759687', u'message': u'Test MountLustreFilesystemsJob (lotus-59vm19.lotus.hpdd.lab.intel.com)', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/25/', u'id': 25, u'logs': u''}
2017-08-21 22:26:57,719 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/87/'], u'complete': True, u'created_at': u'2017-08-22T05:26:54.759687', u'message': u'Test MountLustreFilesystemsJob (lotus-59vm19.lotus.hpdd.lab.intel.com)', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/25/', u'id': 25, u'logs': u''}
2017-08-21 22:26:57,752 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:26:58,788 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:26:59,829 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:00,871 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:01,912 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:02,951 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:03,992 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:05,027 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:06,065 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:07,102 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:08,145 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:09,187 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:10,231 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:11,270 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:12,309 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:13,349 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:14,390 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:15,430 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:16,470 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:17,512 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:18,554 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:19,592 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:20,635 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:21,681 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:22,723 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:23,764 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:24,799 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:25,838 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:26,878 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:27,917 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:28,956 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:29,995 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:31,036 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:32,081 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:33,122 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:34,161 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:35,204 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:36,247 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:37,289 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:38,333 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:39,376 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:40,417 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:41,454 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:42,486 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:43,526 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:44,553 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:45,590 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:46,621 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:47,664 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:48,704 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:49,742 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:50,787 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:51,829 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:52,870 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:53,913 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:54,953 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:56,001 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:57,042 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:58,079 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:27:59,111 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "mounted"))
 tripped assertion: u'unmounted' != 'mounted'
2017-08-21 22:28:00,476 - test - DEBUG -         self.wait_until_true(lambda: self._get_mount_job(unmount_job_name))
 evaluated to {u'verb': u'Unmount Filesystem(s)', u'confirmation': u'Unmount all associated Lustre filesystems on this worker.', u'class_name': u'UnmountLustreFilesystemsJob', u'display_order': 130, u'args': {u'host_id': 5}, u'display_group': 3, u'long_description': u'Unmount all associated Lustre filesystems on this worker.'}
2017-08-21 22:28:00,971 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/88/'], u'complete': False, u'created_at': u'2017-08-22T05:28:00.111750', u'message': u'Test MountLustreFilesystemsJob (lotus-59vm19.lotus.hpdd.lab.intel.com)', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/26/', u'id': 26, u'logs': u''}
2017-08-21 22:28:03,089 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/88/'], u'complete': True, u'created_at': u'2017-08-22T05:28:00.111750', u'message': u'Test MountLustreFilesystemsJob (lotus-59vm19.lotus.hpdd.lab.intel.com)', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/26/', u'id': 26, u'logs': u''}
2017-08-21 22:28:03,122 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "unmounted"))
 tripped assertion: u'mounted' != 'unmounted'
2017-08-21 22:28:04,164 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "unmounted"))
 tripped assertion: u'mounted' != 'unmounted'
2017-08-21 22:28:05,208 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "unmounted"))
 tripped assertion: u'mounted' != 'unmounted'
2017-08-21 22:28:06,250 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "unmounted"))
 tripped assertion: u'mounted' != 'unmounted'
2017-08-21 22:28:07,292 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "unmounted"))
 tripped assertion: u'mounted' != 'unmounted'
2017-08-21 22:28:08,333 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "unmounted"))
 tripped assertion: u'mounted' != 'unmounted'
2017-08-21 22:28:09,376 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "unmounted"))
 tripped assertion: u'mounted' != 'unmounted'
2017-08-21 22:28:10,417 - test - DEBUG -         self.wait_for_assert(lambda: self.assertEqual(self.get_json_by_uri(mount['resource_uri'])['state'], "unmounted"))
 tripped assertion: u'mounted' != 'unmounted'
2017-08-21 22:28:11,451 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:11,451 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:11,853 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 22:28:11,853 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:12,231 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:12,231 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:12,631 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 22:28:12,632 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:13,006 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:13,007 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:13,399 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 22:28:13,400 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:13,771 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:13,772 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:14,168 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 22:28:14,169 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:14,551 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:14,551 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:14,955 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 22:28:14,955 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:15,334 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:15,335 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:15,705 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_mount_and_unmount (tests.integration.shared_storage_configuration.test_client_mount_management.TestClientMountManagement) ====="
2017-08-21 22:28:15,706 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:16,778 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:28:16,779 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:16,779 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:17,179 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 22:28:17,179 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:28:17,179 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:17,584 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:28:17,585 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:17,585 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:17,989 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 22:28:17,989 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:28:17,990 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:18,376 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:28:18,377 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:18,377 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:18,749 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 22:28:18,749 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:28:18,750 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:19,121 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:28:19,121 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:19,121 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:19,502 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 22:28:19,502 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:28:19,502 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:19,894 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:28:19,894 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:19,895 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:20,297 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 22:28:20,297 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:28:20,298 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:20,681 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:28:20,682 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:28:20,682 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:21,049 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 22:28:21,049 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:28:21,049 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:21,430 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:28:21,430 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:22,138 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 22:28:22,138 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:22,834 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:28:22,835 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:23,531 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:28:23,532 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:23,938 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 22:28:23,938 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:24,314 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:28:24,314 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:24,687 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:28:24,688 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:25,060 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 22:28:25,060 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:25,447 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:28:25,448 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:28:25,831 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 22:28:51,109 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 22:28:52,738 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 22:28:54,386 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 22:30:10,419 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 22:30:12,357 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 22:30:12,995 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 22:30:12,995 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 22:30:12,996 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 22:30:16,842 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:30:16,842 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:17,231 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:30:17,231 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:18,761 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:30:18,761 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:19,162 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:30:19,162 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:19,688 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:30:19,689 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:20,088 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:30:20,088 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:20,613 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:30:20,613 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:21,013 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:30:21,013 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:21,553 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:30:21,553 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:21,929 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:30:21,929 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:22,469 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:30:22,469 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:22,844 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:30:22,844 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:23,218 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 22:30:23,219 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:23,611 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 22:30:23,611 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:32,710 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 22:30:32,711 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:34,027 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 22:30:34,028 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:35,286 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:30:35,286 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:35,677 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 22:30:35,678 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:36,053 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 22:30:36,053 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:36,505 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 22:30:36,506 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:36,897 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 22:30:36,897 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:30:37,293 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 22:30:37,294 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:31:04,549 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 22:31:04,550 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:31:05,809 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 22:31:05,810 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:31:07,070 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:31:07,071 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:31:07,446 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 22:31:07,446 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:31:07,818 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 22:31:07,819 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:31:08,255 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 22:31:08,256 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:31:08,646 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 22:31:08,647 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:31:09,040 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 22:31:09,040 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:36,356 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 22:36:36,357 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:37,691 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 22:36:37,691 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:38,991 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:36:38,992 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:39,368 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 22:36:39,369 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:39,743 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 22:36:39,743 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:40,181 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 22:36:40,181 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:40,559 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 22:36:40,559 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:40,935 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 22:36:40,935 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:44,113 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 22:36:44,114 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:45,296 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 22:36:45,297 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:46,654 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:36:46,655 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:47,090 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 22:36:47,090 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:47,468 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 22:36:47,468 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:47,902 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 22:36:47,902 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:48,287 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 22:36:48,287 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 22:36:48,287 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:36:48,288 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:48,695 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:36:48,695 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:49,098 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:36:49,098 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:49,471 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:36:49,471 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:49,875 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:36:49,875 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:50,280 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:36:50,280 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:50,650 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:36:50,650 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:51,054 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:36:51,054 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:51,431 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:36:51,431 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:51,835 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:36:51,835 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:52,234 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:36:52,234 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:52,612 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:36:52,612 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:52,986 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:36:52,986 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:53,390 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:36:53,390 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:53,791 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:36:53,792 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:54,195 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:36:54,195 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:54,598 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:36:54,598 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:55,001 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:36:55,001 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:55,377 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 22:36:55,377 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:55,781 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 22:36:55,781 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:56,183 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 22:36:56,184 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:56,586 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 22:36:56,586 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:56,962 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:36:56,963 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:57,359 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:36:57,359 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:57,744 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:36:57,744 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:58,148 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:36:58,148 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:58,542 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:36:58,542 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:58,927 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:36:58,927 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:36:59,302 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:36:59,302 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:36:59,303 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:36:59,304 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:36:59,307 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:36:59,308 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:36:59,706 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:36:59,782 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:36:59,796 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:36:59,802 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:36:59,811 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:36:59,862 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:36:59,863 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:36:59,864 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:36:59,865 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:36:59,866 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:36:59,865 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:36:59,866 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:37:00,245 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:37:00,307 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:37:00,308 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:37:00,318 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:37:00,418 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:37:00,430 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:37:00,431 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 22:37:00,432 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 22:37:00,434 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 22:37:00,435 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 22:37:00,753 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 22:37:00,765 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 22:37:00,810 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 22:37:00,926 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 22:37:00,928 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:37:00,928 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:37:00,928 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:37:00,929 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:37:01,236 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 22:37:01,335 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 22:37:01,426 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 22:37:01,441 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 22:37:01,442 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 22:37:02,861 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 22:37:02,861 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:37:03,401 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 22:37:03,402 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:37:03,403 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:37:03,403 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:37:03,404 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:37:03,721 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 22:37:03,789 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 22:37:03,848 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 22:37:03,916 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 22:37:03,917 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 22:37:06,395 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 22:37:06,396 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:37:07,004 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 22:37:07,005 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:37:07,006 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:37:07,006 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:37:07,009 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:37:07,311 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 22:37:07,372 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 22:37:07,374 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 22:37:07,494 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 22:37:07,494 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 22:37:10,992 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 22:37:10,993 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:37:11,480 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 22:37:11,480 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 22:37:11,893 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0000
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-MDT0000

2017-08-21 22:37:11,893 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0000
2017-08-21 22:37:12,338 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0000) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0000 exit_status 0 output:
 
2017-08-21 22:37:12,339 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-MDT0000
2017-08-21 22:37:12,681 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-MDT0000) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-MDT0000 exit_status 0 output:
 
2017-08-21 22:37:12,681 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:37:13,004 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 22:37:13,004 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:37:13,423 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 22:37:13,423 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:37:13,852 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 22:37:13,853 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 22:37:14,194 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 22:37:14,194 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 22:37:14,756 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 22:37:14,757 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 22:37:15,185 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 22:37:15,185 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 22:37:15,712 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 22:37:15,713 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 22:37:16,178 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 22:37:16,178 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 22:37:16,790 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 22:37:16,791 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:37:17,213 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 22:37:17,213 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:37:17,510 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 22:37:17,510 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:37:17,798 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 22:37:17,798 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:37:17,798 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:37:18,171 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:37:18,171 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:37:18,573 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:37:18,573 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:37:18,966 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:37:18,966 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:37:19,366 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:37:19,366 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:37:19,761 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:37:19,761 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:37:21,774 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 22:37:23,511 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T05:37:22.543310', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 22:37:30,875 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T05:37:22.543310', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 22:37:30,931 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T05:37:22.681498', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 22:37:30,953 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T05:37:22.681498', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 22:37:32,792 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/'], u'complete': False, u'created_at': u'2017-08-22T05:37:30.854431', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 22:37:40,778 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T05:37:30.854431', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 22:37:40,810 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T05:37:31.539631', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 22:37:40,848 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T05:37:31.539631', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 22:37:41,425 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:42,478 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:43,544 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:44,595 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:45,651 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:46,699 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:47,753 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:48,804 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:49,852 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:50,905 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:51,958 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:53,012 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:54,062 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:55,112 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:56,181 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:57,231 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:58,287 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:37:59,339 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:00,392 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:01,438 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:02,489 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:03,541 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:04,592 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:05,642 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:06,694 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:07,743 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:08,793 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:09,846 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:10,902 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:11,953 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:13,001 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:14,051 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:15,102 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:16,151 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:17,202 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:18,254 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:19,309 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:20,358 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:21,406 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:22,457 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:23,588 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:24,642 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:25,696 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:26,745 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:27,795 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:28,846 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:29,896 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:30,948 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:31,995 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:33,049 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:34,103 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:35,135 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:36,166 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:37,213 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:38,260 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:39,307 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:40,355 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:41,406 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:42,460 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:43,511 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:44,574 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:45,621 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:46,678 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:47,725 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:48,781 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:49,827 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:50,877 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 22:38:52,551 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': False, u'created_at': u'2017-08-22T05:38:51.247394', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 22:49:24,338 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': True, u'created_at': u'2017-08-22T05:38:51.247394', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:49:24,420 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': True, u'created_at': u'2017-08-22T05:38:51.533273', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:49:24,507 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': True, u'created_at': u'2017-08-22T05:38:51.533273', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:49:25,194 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:49:25,194 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:49:25,623 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:49:25,623 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:49:26,690 - test - INFO - Configuring power control on [u'lotus-59vm15.lotus.hpdd.lab.intel.com', u'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 22:49:27,129 - test - DEBUG - Created /api/power_control_type/14/
2017-08-21 22:49:27,217 - test - DEBUG - Created /api/power_control_device/1/
2017-08-21 22:49:27,844 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/1/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm15', u'id': 1, u'resource_uri': u'/api/power_control_device_outlet/1/'}
2017-08-21 22:49:28,528 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/2/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm16', u'id': 2, u'resource_uri': u'/api/power_control_device_outlet/2/'}
2017-08-21 22:49:28,604 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm17', u'id': 3, u'resource_uri': u'/api/power_control_device_outlet/3/'}
2017-08-21 22:49:28,681 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm18', u'id': 4, u'resource_uri': u'/api/power_control_device_outlet/4/'}
2017-08-21 22:49:28,759 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm19', u'id': 5, u'resource_uri': u'/api/power_control_device_outlet/5/'}
2017-08-21 22:49:28,759 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:49:28,759 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:49:31,308 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: []
2017-08-21 22:49:31,308 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:49:31,308 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:49:33,902 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: []
2017-08-21 22:49:34,903 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:49:34,904 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:49:37,531 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 22:49:37,531 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 22:49:37,532 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:49:40,122 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 22:49:41,875 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/39/', u'volume_nodes': [{u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/4/', u'id': 4, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/13/', u'volume_nodes': [{u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/6/', u'id': 6, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/41/', u'volume_nodes': [{u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/10/', u'id': 10, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/13/', u'id': 13, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/37/', u'volume_nodes': [{u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/16/', u'volume_nodes': [{u'use': True, u'volume_id': 5, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/7/', u'id': 7, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/5/', u'id': 5, u'size': u'10737418240'}]'
2017-08-21 22:49:41,876 - test - DEBUG -         self.wait_until_true(lambda: len(self.get_shared_volumes(required_hosts)) >= expected_volumes)
 evaluated to True
2017-08-21 22:49:43,568 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/39/', u'volume_nodes': [{u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/4/', u'id': 4, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/13/', u'volume_nodes': [{u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/6/', u'id': 6, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/41/', u'volume_nodes': [{u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/10/', u'id': 10, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/13/', u'id': 13, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/37/', u'volume_nodes': [{u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/16/', u'volume_nodes': [{u'use': True, u'volume_id': 5, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/7/', u'id': 7, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/5/', u'id': 5, u'size': u'10737418240'}]'
2017-08-21 22:49:45,254 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/30/', u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/', u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/'], u'complete': False, u'created_at': u'2017-08-22T05:49:43.594341', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\n'}
2017-08-21 22:52:47,659 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/30/', u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/', u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/', u'/api/job/49/'], u'complete': True, u'created_at': u'2017-08-22T05:49:43.594341', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:52:47,660 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 22:52:47,660 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:48,104 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 22:52:48,104 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:48,845 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mkdir -p /mnt/testfs
2017-08-21 22:52:48,845 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:49,549 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount -t lustre 10.14.83.84@tcp0:10.14.83.85@tcp0:/testfs /mnt/testfs
2017-08-21 22:52:49,550 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:51,181 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:52:51,181 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:51,889 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/llite/*/max_cached_mb
2017-08-21 22:52:51,889 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:52,614 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/lov/testfs-MDT0000-mdtlov/stripesize
2017-08-21 22:52:52,614 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:53,015 - test - DEBUG -             self.wait_until_true(lambda: "2097152" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/lov/testfs-MDT0000-mdtlov/stripesize"))
 evaluated to True
2017-08-21 22:52:53,015 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/obdfilter/testfs-OST0000/sync_journal
2017-08-21 22:52:53,016 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:53,417 - test - DEBUG -             self.wait_until_true(lambda: "0" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/obdfilter/testfs-OST0000/sync_journal"))
 evaluated to True
2017-08-21 22:52:53,418 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal
2017-08-21 22:52:53,418 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:53,804 - test - DEBUG -             self.wait_until_true(lambda: "1" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal"))
 evaluated to False
2017-08-21 22:52:53,815 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal
2017-08-21 22:52:53,815 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:54,206 - test - DEBUG -             self.wait_until_true(lambda: "1" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal"))
 evaluated to False
2017-08-21 22:52:54,306 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal
2017-08-21 22:52:54,306 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:54,708 - test - DEBUG -             self.wait_until_true(lambda: "1" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal"))
 evaluated to True
2017-08-21 22:52:54,708 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:52:54,709 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:55,412 - test - DEBUG - Unmounting testfs
2017-08-21 22:52:55,413 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount /mnt/testfs
2017-08-21 22:52:55,413 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:56,111 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:52:56,112 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:56,818 - test - DEBUG - `Mount`: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=929624k,nr_inodes=232406,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/vg_00-lv_root on / type ext4 (rw,relatime,seclabel,data=ordered)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/vda1 on /boot type ext3 (rw,relatime,seclabel,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/etc/auto.misc on /misc type autofs (rw,relatime,fd=7,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
-hosts on /net type autofs (rw,relatime,fd=13,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
auto.direct on /root/lab type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /scratch type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /root/chef type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.home on /home type autofs (rw,relatime,fd=25,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=188372k,mode=700)

2017-08-21 22:52:57,175 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:52:57,176 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:57,878 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 22:52:57,878 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:58,571 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:52:58,571 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:59,267 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:52:59,267 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:52:59,673 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 22:52:59,674 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:53:00,050 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:53:00,050 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:53:00,427 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:53:00,427 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:53:00,804 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 22:53:00,804 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:53:01,178 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:53:01,178 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:53:01,926 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/50/', u'/api/job/51/', u'/api/job/52/', u'/api/job/53/', u'/api/job/54/', u'/api/job/55/', u'/api/job/56/', u'/api/job/57/', u'/api/job/58/'], u'complete': False, u'created_at': u'2017-08-22T05:53:00.842548', u'message': u'Remove file system testfs from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/13/', u'id': 13, u'logs': u''}
2017-08-21 22:54:53,249 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/50/', u'/api/job/51/', u'/api/job/52/', u'/api/job/53/', u'/api/job/54/', u'/api/job/55/', u'/api/job/56/', u'/api/job/57/', u'/api/job/58/'], u'complete': True, u'created_at': u'2017-08-22T05:53:00.842548', u'message': u'Remove file system testfs from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/13/', u'id': 13, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:54:53,567 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/59/', u'/api/job/60/'], u'complete': False, u'created_at': u'2017-08-22T05:54:52.705121', u'message': u'Remove target MGS from the filesystem\n*** WARNING *** this is destructive to your filesystem and irreversible', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/14/', u'id': 14, u'logs': u''}
2017-08-21 22:56:00,728 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/59/', u'/api/job/60/'], u'complete': True, u'created_at': u'2017-08-22T05:54:52.705121', u'message': u'Remove target MGS from the filesystem\n*** WARNING *** this is destructive to your filesystem and irreversible', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/14/', u'id': 14, u'logs': u'\n\n\n\n'}
2017-08-21 22:56:02,144 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/61/', u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/'], u'complete': False, u'created_at': u'2017-08-22T05:56:00.853472', u'message': u'Remove host lotus-59vm15.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/15/', u'id': 15, u'logs': u'\n\n\n'}
2017-08-21 22:56:30,265 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/61/', u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/'], u'complete': True, u'created_at': u'2017-08-22T05:56:00.853472', u'message': u'Remove host lotus-59vm15.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/15/', u'id': 15, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:56:30,330 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/70/', u'/api/job/71/', u'/api/job/72/', u'/api/job/73/', u'/api/job/74/', u'/api/job/75/', u'/api/job/76/', u'/api/job/77/', u'/api/job/78/'], u'complete': False, u'created_at': u'2017-08-22T05:56:01.121935', u'message': u'Remove host lotus-59vm16.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/16/', u'id': 16, u'logs': u'\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:56:37,937 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/70/', u'/api/job/71/', u'/api/job/72/', u'/api/job/73/', u'/api/job/74/', u'/api/job/75/', u'/api/job/76/', u'/api/job/77/', u'/api/job/78/'], u'complete': True, u'created_at': u'2017-08-22T05:56:01.121935', u'message': u'Remove host lotus-59vm16.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/16/', u'id': 16, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 22:56:38,087 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:38,087 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:38,488 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:38,488 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:38,865 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:38,865 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:39,270 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:39,271 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:39,636 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:39,636 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:40,040 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:40,040 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:40,415 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:40,415 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:40,818 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:40,819 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:41,206 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:41,206 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:41,602 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:41,603 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:41,993 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:41,994 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:42,364 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_creation_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:42,364 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:43,105 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:56:43,106 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:43,107 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:43,510 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 22:56:43,510 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:43,511 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:43,891 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:56:43,891 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:43,891 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:44,289 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 22:56:44,290 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:44,290 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:44,678 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:56:44,678 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:44,679 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:45,082 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 22:56:45,082 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:45,082 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:45,499 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:56:45,499 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:45,500 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:45,890 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 22:56:45,891 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:45,891 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:46,275 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:56:46,275 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:46,276 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:46,658 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 22:56:46,658 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:46,659 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:47,035 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 22:56:47,035 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:56:47,036 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:47,424 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 22:56:47,424 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 22:56:47,424 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:47,801 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:56:47,801 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:48,495 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 22:56:48,495 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:49,190 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:56:49,191 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:49,889 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:56:49,890 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:50,288 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 22:56:50,289 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:50,674 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:56:50,674 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:51,046 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:56:51,047 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:51,418 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 22:56:51,418 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:51,794 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 22:56:51,794 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:56:52,178 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 22:57:17,484 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 22:57:19,063 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 22:57:20,723 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 22:58:36,432 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 22:58:38,385 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 22:58:39,085 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 22:58:39,085 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 22:58:39,085 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 22:58:43,094 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:58:43,094 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:43,493 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:58:43,494 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:43,947 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:58:43,947 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:44,349 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:58:44,349 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:44,784 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:58:44,784 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:45,186 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:58:45,186 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:45,563 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:58:45,564 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:45,938 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:58:45,938 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:46,329 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:58:46,329 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:46,732 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:58:46,733 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:47,108 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 22:58:47,109 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:47,489 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 22:58:47,489 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:47,878 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 22:58:47,879 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:48,275 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 22:58:48,275 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:48,815 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 22:58:48,815 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:50,134 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 22:58:50,134 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:51,341 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:58:51,341 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:51,730 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 22:58:51,731 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:52,115 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 22:58:52,116 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:52,541 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 22:58:52,541 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:52,924 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 22:58:52,925 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:53,326 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 22:58:53,326 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:53,862 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 22:58:53,862 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:55,129 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 22:58:55,130 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:56,337 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:58:56,338 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:56,720 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 22:58:56,721 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:57,101 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 22:58:57,101 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:57,546 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 22:58:57,546 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:57,935 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 22:58:57,935 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:58,339 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 22:58:58,339 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:58:58,829 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 22:58:58,829 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:00,120 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 22:59:00,121 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:01,489 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:59:01,489 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:01,863 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 22:59:01,863 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:02,244 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 22:59:02,244 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:02,667 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 22:59:02,668 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:03,041 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 22:59:03,041 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:03,419 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 22:59:03,420 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:03,905 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 22:59:03,906 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:05,207 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 22:59:05,207 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:06,500 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 22:59:06,501 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:06,902 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 22:59:06,902 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:07,262 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 22:59:07,263 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:07,664 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 22:59:07,664 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:08,037 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 22:59:08,038 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 22:59:08,038 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:59:08,038 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:08,440 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:59:08,440 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:08,786 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:59:08,786 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:09,158 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:59:09,159 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:09,526 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:59:09,527 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:09,928 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 22:59:09,928 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:10,316 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:59:10,317 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:10,708 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:59:10,709 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:11,105 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:59:11,105 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:11,500 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:59:11,501 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:11,905 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:59:11,906 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:12,308 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 22:59:12,308 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:12,692 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:59:12,692 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:13,096 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:59:13,097 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:13,499 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:59:13,499 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:13,879 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:59:13,880 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:14,280 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:59:14,280 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:14,682 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 22:59:14,683 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:15,057 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 22:59:15,058 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:15,463 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 22:59:15,463 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:15,871 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 22:59:15,871 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:16,273 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 22:59:16,273 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:16,675 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:59:16,675 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:17,078 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:59:17,079 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:17,479 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:59:17,479 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:17,742 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:59:17,743 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:18,145 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:59:18,145 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:18,542 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 22:59:18,542 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:18,926 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:59:18,927 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:59:18,929 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:59:18,931 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:59:18,934 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:59:18,936 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 22:59:19,324 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:19,338 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:19,455 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:19,513 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:19,526 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:19,528 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:19,529 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:59:19,530 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:59:19,531 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:59:19,532 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:59:19,535 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:59:19,537 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 22:59:19,940 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:19,982 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:20,048 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:20,049 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:20,099 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:20,101 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 22:59:20,102 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 22:59:20,103 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 22:59:20,103 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 22:59:20,104 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 22:59:20,466 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 22:59:20,481 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 22:59:20,586 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 22:59:20,590 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 22:59:20,591 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:59:20,592 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:59:20,593 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:59:20,594 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:59:20,867 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 22:59:20,951 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 22:59:20,976 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 22:59:20,978 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 22:59:20,978 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 22:59:22,370 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 22:59:22,370 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:59:22,924 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 22:59:22,925 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:59:22,926 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:59:22,927 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:59:22,928 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:59:23,289 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 22:59:23,314 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 22:59:23,327 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 22:59:23,402 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 22:59:23,403 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 22:59:25,903 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 22:59:25,903 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:59:26,536 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 22:59:26,537 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:59:26,537 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:59:26,538 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:59:26,539 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:59:26,868 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 22:59:26,869 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 22:59:26,929 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 22:59:26,933 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 22:59:26,934 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 22:59:30,399 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 22:59:30,399 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:59:30,914 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 22:59:30,914 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 22:59:31,197 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0001
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS

2017-08-21 22:59:31,197 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0001
2017-08-21 22:59:31,502 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0001) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0001 exit_status 0 output:
 
2017-08-21 22:59:31,503 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000
2017-08-21 22:59:31,923 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000 exit_status 0 output:
 
2017-08-21 22:59:31,924 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS
2017-08-21 22:59:32,403 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS exit_status 0 output:
 
2017-08-21 22:59:32,403 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:59:32,709 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 22:59:32,710 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:59:33,018 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 22:59:33,018 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:59:33,434 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 22:59:33,434 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 22:59:33,878 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 22:59:33,878 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 22:59:34,390 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 22:59:34,391 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 22:59:34,739 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 22:59:34,739 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 22:59:35,336 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 22:59:35,337 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 22:59:35,822 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 22:59:35,822 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 22:59:36,378 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 22:59:36,379 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 22:59:36,663 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 22:59:36,663 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 22:59:36,952 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 22:59:36,952 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 22:59:37,353 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 22:59:37,354 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:59:37,354 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:37,730 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:59:37,730 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:38,126 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:59:38,126 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:38,523 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:59:38,524 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:38,899 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:59:38,900 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:39,301 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 22:59:39,301 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 22:59:41,796 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 22:59:42,804 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T05:59:41.208919', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 22:59:49,819 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T05:59:41.208919', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 22:59:50,883 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': False, u'created_at': u'2017-08-22T05:59:49.788596', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 23:03:18,933 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T05:59:49.788596', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 23:03:19,810 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': False, u'created_at': u'2017-08-22T06:03:18.699293', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 23:14:14,831 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/', u'/api/job/4/', u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/'], u'complete': True, u'created_at': u'2017-08-22T06:03:18.699293', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:14:15,213 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:14:15,213 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:14:15,985 - test - INFO - Configuring power control on [u'lotus-59vm15.lotus.hpdd.lab.intel.com']
2017-08-21 23:14:16,414 - test - DEBUG - Created /api/power_control_type/14/
2017-08-21 23:14:16,494 - test - DEBUG - Created /api/power_control_device/1/
2017-08-21 23:14:17,019 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/1/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm15', u'id': 1, u'resource_uri': u'/api/power_control_device_outlet/1/'}
2017-08-21 23:14:17,080 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm16', u'id': 2, u'resource_uri': u'/api/power_control_device_outlet/2/'}
2017-08-21 23:14:17,147 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm17', u'id': 3, u'resource_uri': u'/api/power_control_device_outlet/3/'}
2017-08-21 23:14:17,210 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm18', u'id': 4, u'resource_uri': u'/api/power_control_device_outlet/4/'}
2017-08-21 23:14:17,278 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm19', u'id': 5, u'resource_uri': u'/api/power_control_device_outlet/5/'}
2017-08-21 23:14:17,279 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 23:14:17,279 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:14:19,880 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: []
2017-08-21 23:14:20,882 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 23:14:20,882 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:14:23,489 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: []
2017-08-21 23:14:24,490 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 23:14:24,490 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:14:27,102 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com']
2017-08-21 23:14:29,609 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/'], u'complete': False, u'created_at': u'2017-08-22T06:14:28.322650', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u'\n\n'}
2017-08-21 23:16:32,749 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/'], u'complete': True, u'created_at': u'2017-08-22T06:14:28.322650', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:16:32,750 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 23:16:32,751 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:16:33,496 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mkdir -p /mnt/testfs
2017-08-21 23:16:33,496 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:16:34,204 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount -t lustre 10.14.83.84@tcp0:/testfs /mnt/testfs
2017-08-21 23:16:34,205 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:00,004 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:17:00,006 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:00,699 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/llite/*/max_cached_mb
2017-08-21 23:17:00,699 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:01,393 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:17:01,393 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:02,094 - test - DEBUG - Unmounting testfs
2017-08-21 23:17:02,095 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount /mnt/testfs
2017-08-21 23:17:02,095 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:02,789 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:17:02,789 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:03,483 - test - DEBUG - `Mount`: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=929624k,nr_inodes=232406,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/vg_00-lv_root on / type ext4 (rw,relatime,seclabel,data=ordered)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/vda1 on /boot type ext3 (rw,relatime,seclabel,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/etc/auto.misc on /misc type autofs (rw,relatime,fd=7,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
-hosts on /net type autofs (rw,relatime,fd=13,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
auto.direct on /root/lab type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /scratch type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /root/chef type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.home on /home type autofs (rw,relatime,fd=25,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=188372k,mode=700)

2017-08-21 23:17:04,166 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/28/'], u'complete': False, u'created_at': u'2017-08-22T06:17:03.098173', u'message': u'Updating configuration parameters', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u''}
2017-08-21 23:17:05,219 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/28/'], u'complete': True, u'created_at': u'2017-08-22T06:17:03.098173', u'message': u'Updating configuration parameters', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n'}
2017-08-21 23:17:05,220 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mkdir -p /mnt/testfs
2017-08-21 23:17:05,220 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:05,922 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount -t lustre 10.14.83.84@tcp0:/testfs /mnt/testfs
2017-08-21 23:17:05,922 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:06,671 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:17:06,672 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:07,400 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/llite/*/max_cached_mb
2017-08-21 23:17:07,401 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:08,053 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:17:08,053 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:08,751 - test - DEBUG - Unmounting testfs
2017-08-21 23:17:08,751 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount /mnt/testfs
2017-08-21 23:17:08,752 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:09,400 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:17:09,401 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:10,098 - test - DEBUG - `Mount`: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=929624k,nr_inodes=232406,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/vg_00-lv_root on / type ext4 (rw,relatime,seclabel,data=ordered)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/vda1 on /boot type ext3 (rw,relatime,seclabel,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/etc/auto.misc on /misc type autofs (rw,relatime,fd=7,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
-hosts on /net type autofs (rw,relatime,fd=13,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
auto.direct on /root/lab type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /scratch type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /root/chef type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.home on /home type autofs (rw,relatime,fd=25,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=188372k,mode=700)

2017-08-21 23:17:10,370 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:17:10,370 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:11,069 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:17:11,069 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:11,766 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:17:11,767 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:12,459 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:17:12,459 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:12,865 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:17:12,866 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:13,246 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:17:13,246 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:13,626 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:17:13,627 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:14,001 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:17:14,001 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:14,375 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:17:14,375 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:17:15,074 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/29/', u'/api/job/30/', u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/'], u'complete': False, u'created_at': u'2017-08-22T06:17:14.044992', u'message': u'Remove file system testfs from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u'\n'}
2017-08-21 23:18:24,593 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/29/', u'/api/job/30/', u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/'], u'complete': True, u'created_at': u'2017-08-22T06:17:14.044992', u'message': u'Remove file system testfs from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u'\n\n\n\n\n\n'}
2017-08-21 23:18:24,887 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/36/', u'/api/job/37/'], u'complete': False, u'created_at': u'2017-08-22T06:18:24.028301', u'message': u'Remove target MGS from the filesystem\n*** WARNING *** this is destructive to your filesystem and irreversible', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-21 23:19:15,106 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/36/', u'/api/job/37/'], u'complete': True, u'created_at': u'2017-08-22T06:18:24.028301', u'message': u'Remove target MGS from the filesystem\n*** WARNING *** this is destructive to your filesystem and irreversible', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u'\n\n'}
2017-08-21 23:19:15,877 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/', u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/'], u'complete': False, u'created_at': u'2017-08-22T06:19:14.852813', u'message': u'Remove host lotus-59vm15.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u'\n\n\n'}
2017-08-21 23:19:47,177 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/', u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/'], u'complete': True, u'created_at': u'2017-08-22T06:19:14.852813', u'message': u'Remove host lotus-59vm15.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:19:47,311 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:47,311 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:47,710 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:47,711 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:48,093 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:48,093 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:48,496 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:48,496 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:48,865 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:48,865 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:49,252 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:49,252 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:49,628 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:49,628 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:50,033 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:50,034 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:50,406 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:50,406 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:50,807 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:50,807 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:51,195 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:51,195 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:51,570 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_update_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:51,570 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:52,344 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 23:19:52,344 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:52,345 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:52,747 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 23:19:52,748 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:52,748 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:53,149 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 23:19:53,149 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:53,150 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:53,553 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 23:19:53,553 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:53,553 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:53,929 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 23:19:53,930 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:53,930 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:54,307 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-21 23:19:54,307 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:54,307 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:54,690 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 23:19:54,691 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:54,691 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:55,092 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-21 23:19:55,092 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:55,093 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:55,470 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 23:19:55,470 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:55,470 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:55,868 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 23:19:55,869 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:55,869 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:56,241 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 23:19:56,241 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:19:56,241 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:56,614 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-21 23:19:56,614 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:19:56,614 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:56,989 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:19:56,989 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:57,695 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:19:57,695 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:58,392 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:19:58,392 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:59,098 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:19:59,098 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:59,491 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:19:59,491 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:19:59,866 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:19:59,866 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:20:00,241 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:20:00,241 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:20:00,616 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:20:00,616 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:20:00,998 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:20:00,998 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:20:01,371 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 23:20:26,680 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 23:20:28,268 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 23:20:29,933 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 23:21:44,068 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 23:21:46,065 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 23:21:46,576 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 23:21:46,576 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 23:21:46,576 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 23:21:50,432 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:21:50,432 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:50,838 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:21:50,838 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:51,274 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:21:51,275 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:51,677 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:21:51,677 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:52,050 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:21:52,051 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:52,456 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:21:52,456 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:52,833 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:21:52,833 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:53,236 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:21:53,236 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:53,611 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:21:53,611 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:54,008 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:21:54,008 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:54,400 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:21:54,400 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:54,774 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:21:54,774 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:55,148 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 23:21:55,149 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:55,552 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 23:21:55,553 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:56,090 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 23:21:56,090 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:57,422 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 23:21:57,423 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:58,682 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:21:58,682 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:59,075 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 23:21:59,075 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:59,460 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 23:21:59,460 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:21:59,905 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 23:21:59,905 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:00,292 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 23:22:00,292 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:00,688 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 23:22:00,688 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:01,186 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 23:22:01,186 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:02,427 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 23:22:02,428 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:03,678 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:22:03,679 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:04,062 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 23:22:04,062 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:04,436 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 23:22:04,437 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:04,853 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 23:22:04,854 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:05,223 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 23:22:05,223 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:05,626 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 23:22:05,626 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:06,118 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 23:22:06,119 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:07,443 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 23:22:07,443 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:08,745 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:22:08,745 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:09,114 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 23:22:09,115 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:09,499 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 23:22:09,499 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:09,922 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 23:22:09,922 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:10,306 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 23:22:10,307 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:10,710 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 23:22:10,711 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:11,193 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 23:22:11,194 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:12,437 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 23:22:12,438 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:13,694 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:22:13,695 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:14,084 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 23:22:14,085 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:14,460 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 23:22:14,461 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:14,894 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 23:22:14,894 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:15,272 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 23:22:15,272 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-21 23:22:15,273 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 23:22:15,273 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:15,677 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 23:22:15,678 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:16,080 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 23:22:16,080 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:16,466 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 23:22:16,466 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:16,871 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 23:22:16,871 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:17,272 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 23:22:17,273 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:17,648 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:22:17,648 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:18,052 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:22:18,052 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:18,455 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:22:18,455 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:18,856 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:22:18,857 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:19,255 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:22:19,255 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:19,654 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:22:19,655 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:20,042 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 23:22:20,042 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:20,447 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 23:22:20,448 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:20,850 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 23:22:20,850 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:21,241 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 23:22:21,241 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:21,617 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 23:22:21,617 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:22,018 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 23:22:22,018 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:22,391 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 23:22:22,392 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:22,793 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 23:22:22,793 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:23,167 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 23:22:23,167 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:23,570 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 23:22:23,570 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:23,967 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:22:23,967 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:24,371 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:22:24,371 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:24,768 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:22:24,768 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:25,142 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:22:25,143 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:25,544 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:22:25,544 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:25,918 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:22:25,918 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:26,298 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 23:22:26,299 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 23:22:26,300 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 23:22:26,302 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 23:22:26,306 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 23:22:26,309 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 23:22:26,685 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:26,778 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:26,799 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:26,803 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:26,809 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:26,853 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:26,854 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 23:22:26,855 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 23:22:26,856 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 23:22:26,859 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 23:22:26,861 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 23:22:26,862 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 23:22:27,174 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:27,182 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:27,185 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:27,292 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:27,347 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:27,370 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:22:27,371 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 23:22:27,372 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 23:22:27,372 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 23:22:27,374 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 23:22:27,790 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 23:22:27,827 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 23:22:27,835 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 23:22:27,911 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 23:22:27,912 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:22:27,913 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:22:27,915 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:22:27,915 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:22:28,236 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 23:22:28,316 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 23:22:28,351 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 23:22:28,356 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 23:22:28,356 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 23:22:29,748 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 23:22:29,748 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:22:30,393 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 23:22:30,394 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:22:30,395 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:22:30,396 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:22:30,397 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:22:30,735 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 23:22:30,749 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 23:22:30,843 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 23:22:30,927 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 23:22:30,928 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 23:22:33,310 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 23:22:33,310 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:22:33,842 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 23:22:33,843 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:22:33,844 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:22:33,845 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:22:33,846 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:22:34,156 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 23:22:34,173 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 23:22:34,289 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 23:22:34,331 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 23:22:34,332 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 23:22:37,816 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 23:22:37,817 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:22:38,333 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 23:22:38,333 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 23:22:38,616 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS

2017-08-21 23:22:38,617 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000
2017-08-21 23:22:38,936 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000 exit_status 0 output:
 
2017-08-21 23:22:38,936 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS
2017-08-21 23:22:39,396 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS exit_status 0 output:
 
2017-08-21 23:22:39,396 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:22:39,704 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 23:22:39,705 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:22:39,987 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 23:22:39,988 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:22:40,288 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 23:22:40,290 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 23:22:40,626 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 23:22:40,626 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 23:22:41,330 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 23:22:41,331 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 23:22:41,683 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 23:22:41,683 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 23:22:42,191 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 23:22:42,192 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 23:22:42,558 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 23:22:42,558 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 23:22:43,236 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 23:22:43,237 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:22:43,672 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 23:22:43,673 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:22:43,985 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 23:22:43,985 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:22:44,424 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 23:22:44,424 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 23:22:44,424 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:44,808 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 23:22:44,808 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:45,209 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 23:22:45,209 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:45,610 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 23:22:45,610 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:46,010 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 23:22:46,010 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:46,407 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 23:22:46,407 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:22:48,441 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 23:22:49,513 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T06:22:48.574992', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 23:22:57,549 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T06:22:48.574992', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 23:22:57,630 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T06:22:48.688059', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 23:22:57,664 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T06:22:48.688059', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 23:22:59,473 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/'], u'complete': False, u'created_at': u'2017-08-22T06:22:57.540095', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 23:23:04,691 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T06:22:57.540095', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 23:23:04,726 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T06:22:58.313790', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 23:23:04,762 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T06:22:58.313790', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 23:23:05,395 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:06,442 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:08,164 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:09,224 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:10,275 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:11,325 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:12,377 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:13,429 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:14,480 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:15,531 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:16,584 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:17,634 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:18,687 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:19,736 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:20,786 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:21,833 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:22,883 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:23,930 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:24,982 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:26,022 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:27,074 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:28,125 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:29,178 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:30,229 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:31,275 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:32,322 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:33,367 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:34,414 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:35,461 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:36,509 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:37,556 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:38,603 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:39,727 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:40,777 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:41,828 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:42,873 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:43,919 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:44,968 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:46,009 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:47,060 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:48,110 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:49,163 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:50,203 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:23:51,909 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': False, u'created_at': u'2017-08-22T06:23:50.584430', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 23:33:23,309 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': True, u'created_at': u'2017-08-22T06:23:50.584430', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:33:23,384 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': False, u'created_at': u'2017-08-22T06:23:50.899970', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:34:14,939 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': True, u'created_at': u'2017-08-22T06:23:50.899970', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:34:15,631 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:34:15,631 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:34:16,032 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:34:16,032 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:34:17,106 - test - INFO - Configuring power control on [u'lotus-59vm15.lotus.hpdd.lab.intel.com', u'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 23:34:17,501 - test - DEBUG - Created /api/power_control_type/14/
2017-08-21 23:34:17,612 - test - DEBUG - Created /api/power_control_device/1/
2017-08-21 23:34:18,234 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/1/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm15', u'id': 1, u'resource_uri': u'/api/power_control_device_outlet/1/'}
2017-08-21 23:34:18,949 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/2/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm16', u'id': 2, u'resource_uri': u'/api/power_control_device_outlet/2/'}
2017-08-21 23:34:19,021 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm17', u'id': 3, u'resource_uri': u'/api/power_control_device_outlet/3/'}
2017-08-21 23:34:19,093 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm18', u'id': 4, u'resource_uri': u'/api/power_control_device_outlet/4/'}
2017-08-21 23:34:19,177 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm19', u'id': 5, u'resource_uri': u'/api/power_control_device_outlet/5/'}
2017-08-21 23:34:19,178 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 23:34:19,178 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:34:21,835 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: []
2017-08-21 23:34:21,835 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 23:34:21,836 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:34:24,490 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: []
2017-08-21 23:34:25,492 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 23:34:25,492 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:34:28,111 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 23:34:28,112 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-21 23:34:28,112 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:34:30,726 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-21 23:34:32,481 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/11/', u'volume_nodes': [{u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/39/', u'volume_nodes': [{u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/5/', u'id': 5, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/13/', u'id': 13, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/35/', u'volume_nodes': [{u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/11/', u'id': 11, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/14/', u'volume_nodes': [{u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/10/', u'id': 10, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/52/', u'volume_nodes': [{u'use': True, u'volume_id': 8, u'primary': True, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 8, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/12/', u'id': 12, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/8/', u'id': 8, u'size': u'10672993730'}]'
2017-08-21 23:34:32,482 - test - DEBUG -         self.wait_until_true(lambda: len(self.get_shared_volumes(required_hosts)) >= expected_volumes)
 evaluated to True
2017-08-21 23:34:34,229 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/11/', u'volume_nodes': [{u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/39/', u'volume_nodes': [{u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/5/', u'id': 5, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/13/', u'id': 13, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/35/', u'volume_nodes': [{u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/11/', u'id': 11, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/14/', u'volume_nodes': [{u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/10/', u'id': 10, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/52/', u'volume_nodes': [{u'use': True, u'volume_id': 8, u'primary': True, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 8, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/12/', u'id': 12, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/8/', u'id': 8, u'size': u'10672993730'}]'
2017-08-21 23:34:35,919 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/29/', u'/api/job/30/', u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/', u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/'], u'complete': False, u'created_at': u'2017-08-22T06:34:34.252097', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u'\n\n\n'}
2017-08-21 23:38:24,572 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/29/', u'/api/job/30/', u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/', u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/48/'], u'complete': True, u'created_at': u'2017-08-22T06:34:34.252097', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:38:24,572 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 23:38:24,573 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:25,037 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-21 23:38:25,038 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:25,769 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mkdir -p /mnt/testfs
2017-08-21 23:38:25,770 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:26,472 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount -t lustre 10.14.83.84@tcp0:10.14.83.85@tcp0:/testfs /mnt/testfs
2017-08-21 23:38:26,473 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:27,214 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:38:27,215 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:27,918 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/llite/*/max_cached_mb
2017-08-21 23:38:27,918 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:28,617 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/lov/testfs-MDT0000-mdtlov/stripesize
2017-08-21 23:38:28,618 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:29,018 - test - DEBUG -             self.wait_until_true(lambda: "2097152" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/lov/testfs-MDT0000-mdtlov/stripesize"))
 evaluated to True
2017-08-21 23:38:29,018 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/obdfilter/testfs-OST0000/sync_journal
2017-08-21 23:38:29,019 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:29,408 - test - DEBUG -             self.wait_until_true(lambda: "0" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/obdfilter/testfs-OST0000/sync_journal"))
 evaluated to True
2017-08-21 23:38:29,408 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal
2017-08-21 23:38:29,409 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:29,785 - test - DEBUG -             self.wait_until_true(lambda: "1" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal"))
 evaluated to True
2017-08-21 23:38:29,786 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:38:29,794 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:30,492 - test - DEBUG - Unmounting testfs
2017-08-21 23:38:30,492 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount /mnt/testfs
2017-08-21 23:38:30,493 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:31,188 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:38:31,188 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:38:31,892 - test - DEBUG - `Mount`: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=929624k,nr_inodes=232406,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/vg_00-lv_root on / type ext4 (rw,relatime,seclabel,data=ordered)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/vda1 on /boot type ext3 (rw,relatime,seclabel,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/etc/auto.misc on /misc type autofs (rw,relatime,fd=7,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
-hosts on /net type autofs (rw,relatime,fd=13,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
auto.direct on /root/lab type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /scratch type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /root/chef type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.home on /home type autofs (rw,relatime,fd=25,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=188372k,mode=700)

2017-08-21 23:38:32,287 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/', u'/api/job/53/', u'/api/job/54/', u'/api/job/55/'], u'complete': False, u'created_at': u'2017-08-22T06:38:31.180386', u'message': u'Test writeconf', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n'}
2017-08-21 23:41:41,290 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/', u'/api/job/53/', u'/api/job/54/', u'/api/job/55/'], u'complete': True, u'created_at': u'2017-08-22T06:38:31.180386', u'message': u'Test writeconf', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:41:41,588 - test - DEBUG - set_state /api/target/1/ mounted
2017-08-21 23:41:41,844 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/56/'], u'complete': False, u'created_at': u'2017-08-22T06:41:41.007939', u'message': u'Start target MGS', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/13/', u'id': 13, u'logs': u''}
2017-08-21 23:41:57,520 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/56/', u'/api/job/57/'], u'complete': True, u'created_at': u'2017-08-22T06:41:41.007939', u'message': u'Start target MGS', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/13/', u'id': 13, u'logs': u'\n\n\n\n\n'}
2017-08-21 23:41:57,521 - test - DEBUG - assertValue state /api/target/1/ mounted
2017-08-21 23:41:57,665 - test - DEBUG - set_state /api/target/2/ mounted
2017-08-21 23:41:57,943 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/58/'], u'complete': False, u'created_at': u'2017-08-22T06:41:57.115142', u'message': u'Start target testfs-MDT0000', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/14/', u'id': 14, u'logs': u''}
2017-08-21 23:42:24,034 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/58/'], u'complete': True, u'created_at': u'2017-08-22T06:41:57.115142', u'message': u'Start target testfs-MDT0000', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/14/', u'id': 14, u'logs': u''}
2017-08-21 23:42:24,034 - test - DEBUG - assertValue state /api/target/2/ mounted
2017-08-21 23:42:24,190 - test - DEBUG - set_state /api/filesystem/1/ available
2017-08-21 23:42:24,645 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/59/', u'/api/job/60/', u'/api/job/61/'], u'complete': False, u'created_at': u'2017-08-22T06:42:23.752777', u'message': u'Start file system testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/15/', u'id': 15, u'logs': u''}
2017-08-21 23:42:40,462 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/59/', u'/api/job/60/', u'/api/job/61/'], u'complete': True, u'created_at': u'2017-08-22T06:42:23.752777', u'message': u'Start file system testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/15/', u'id': 15, u'logs': u'\n'}
2017-08-21 23:42:40,463 - test - DEBUG - assertValue state /api/filesystem/1/ available
2017-08-21 23:42:41,059 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mkdir -p /mnt/testfs
2017-08-21 23:42:41,060 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:41,760 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount -t lustre 10.14.83.84@tcp0:10.14.83.85@tcp0:/testfs /mnt/testfs
2017-08-21 23:42:41,760 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:42,505 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:42:42,505 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:43,208 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/llite/*/max_cached_mb
2017-08-21 23:42:43,208 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:43,900 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/lov/testfs-MDT0000-mdtlov/stripesize
2017-08-21 23:42:43,900 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:44,298 - test - DEBUG -             self.wait_until_true(lambda: "2097152" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/lov/testfs-MDT0000-mdtlov/stripesize"))
 evaluated to True
2017-08-21 23:42:44,299 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/obdfilter/testfs-OST0000/sync_journal
2017-08-21 23:42:44,299 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:44,692 - test - DEBUG -             self.wait_until_true(lambda: "0" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/obdfilter/testfs-OST0000/sync_journal"))
 evaluated to True
2017-08-21 23:42:44,693 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal
2017-08-21 23:42:44,693 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:45,102 - test - DEBUG -             self.wait_until_true(lambda: "1" == self.remote_operations.read_proc(server_address, "/proc/fs/lustre/obdfilter/testfs-OST0001/sync_journal"))
 evaluated to True
2017-08-21 23:42:45,102 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:42:45,102 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:45,800 - test - DEBUG - Unmounting testfs
2017-08-21 23:42:45,800 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount /mnt/testfs
2017-08-21 23:42:45,803 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:46,497 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:42:46,497 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:47,196 - test - DEBUG - `Mount`: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=929624k,nr_inodes=232406,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/vg_00-lv_root on / type ext4 (rw,relatime,seclabel,data=ordered)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/vda1 on /boot type ext3 (rw,relatime,seclabel,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/etc/auto.misc on /misc type autofs (rw,relatime,fd=7,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
-hosts on /net type autofs (rw,relatime,fd=13,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
auto.direct on /root/lab type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /scratch type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.direct on /root/chef type autofs (rw,relatime,fd=19,pgrp=1005,timeout=300,minproto=5,maxproto=5,direct)
auto.home on /home type autofs (rw,relatime,fd=25,pgrp=1005,timeout=300,minproto=5,maxproto=5,indirect)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=188372k,mode=700)

2017-08-21 23:42:47,491 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:42:47,492 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:48,208 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:42:48,208 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:48,906 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:42:48,906 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:49,603 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:42:49,604 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:49,982 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:42:49,982 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:50,363 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:42:50,364 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:50,728 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:42:50,728 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:51,114 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:42:51,115 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:51,487 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:42:51,487 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:42:52,168 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/', u'/api/job/70/'], u'complete': False, u'created_at': u'2017-08-22T06:42:51.149446', u'message': u'Remove file system testfs from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/16/', u'id': 16, u'logs': u''}
2017-08-21 23:44:58,380 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/', u'/api/job/70/'], u'complete': True, u'created_at': u'2017-08-22T06:42:51.149446', u'message': u'Remove file system testfs from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/16/', u'id': 16, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:44:58,680 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/71/', u'/api/job/72/'], u'complete': False, u'created_at': u'2017-08-22T06:44:57.811134', u'message': u'Remove target MGS from the filesystem\n*** WARNING *** this is destructive to your filesystem and irreversible', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/17/', u'id': 17, u'logs': u''}
2017-08-21 23:45:49,957 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/71/', u'/api/job/72/'], u'complete': True, u'created_at': u'2017-08-22T06:44:57.811134', u'message': u'Remove target MGS from the filesystem\n*** WARNING *** this is destructive to your filesystem and irreversible', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/17/', u'id': 17, u'logs': u'\n\n\n\n'}
2017-08-21 23:45:51,299 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/73/', u'/api/job/74/', u'/api/job/75/', u'/api/job/76/', u'/api/job/77/', u'/api/job/78/', u'/api/job/79/', u'/api/job/80/', u'/api/job/81/'], u'complete': False, u'created_at': u'2017-08-22T06:45:50.065472', u'message': u'Remove host lotus-59vm15.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/18/', u'id': 18, u'logs': u'\n\n\n'}
2017-08-21 23:46:22,622 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/73/', u'/api/job/74/', u'/api/job/75/', u'/api/job/76/', u'/api/job/77/', u'/api/job/78/', u'/api/job/79/', u'/api/job/80/', u'/api/job/81/'], u'complete': True, u'created_at': u'2017-08-22T06:45:50.065472', u'message': u'Remove host lotus-59vm15.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/18/', u'id': 18, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:46:22,678 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/82/', u'/api/job/83/', u'/api/job/84/', u'/api/job/85/', u'/api/job/86/', u'/api/job/87/', u'/api/job/88/', u'/api/job/89/', u'/api/job/90/'], u'complete': False, u'created_at': u'2017-08-22T06:45:50.285841', u'message': u'Remove host lotus-59vm16.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/19/', u'id': 19, u'logs': u'\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:46:29,190 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/82/', u'/api/job/83/', u'/api/job/84/', u'/api/job/85/', u'/api/job/86/', u'/api/job/87/', u'/api/job/88/', u'/api/job/89/', u'/api/job/90/'], u'complete': True, u'created_at': u'2017-08-22T06:45:50.285841', u'message': u'Remove host lotus-59vm16.lotus.hpdd.lab.intel.com from configuration', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/19/', u'id': 19, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-21 23:46:29,332 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:46:29,333 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:29,735 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:46:29,735 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:30,116 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:46:30,117 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:30,518 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:46:30,519 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:30,904 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:46:30,904 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:31,305 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:46:31,306 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:31,678 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:46:31,679 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:32,317 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:46:32,317 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:32,690 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:46:32,690 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:33,074 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:46:33,075 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:33,448 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:46:33,448 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:33,823 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_writeconf_conf_params (tests.integration.shared_storage_configuration.test_conf_params.TestConfParams) ====="
2017-08-21 23:46:33,823 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:34,564 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 23:46:34,564 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:46:34,564 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:34,967 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-21 23:46:34,967 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_corosync_reverse_dependencies (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-21 23:46:34,968 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:35,361 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-21 23:46:35,362 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:46:35,362 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:35,752 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-21 23:46:35,752 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_corosync_reverse_dependencies (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-21 23:46:35,752 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:36,124 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:46:36,125 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:36,828 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:46:36,828 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:37,521 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:46:37,521 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:38,209 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:46:38,210 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:38,613 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:46:38,613 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:38,996 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:46:38,996 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:39,370 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:46:39,370 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:39,741 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-21 23:46:39,742 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:40,114 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-21 23:46:40,114 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:46:40,489 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-21 23:47:09,743 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-21 23:47:11,313 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-21 23:47:12,958 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-21 23:48:27,912 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-21 23:48:29,799 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-21 23:48:30,375 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-21 23:48:30,375 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-21 23:48:30,375 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-21 23:48:34,272 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:48:34,273 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:34,739 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:48:34,739 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:35,193 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:48:35,193 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:35,590 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:48:35,590 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:36,031 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:48:36,032 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:36,436 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:48:36,436 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:36,812 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:48:36,812 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:37,216 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:48:37,216 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:37,590 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:48:37,591 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:37,987 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:48:37,988 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:38,378 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-21 23:48:38,378 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:38,770 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-21 23:48:38,770 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:39,143 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 23:48:39,143 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:39,548 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 23:48:39,548 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:40,077 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 23:48:40,077 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:41,535 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 23:48:41,536 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:43,003 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:48:43,003 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:43,406 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 23:48:43,406 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:43,831 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 23:48:43,831 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:44,281 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 23:48:44,282 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:44,657 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-21 23:48:44,657 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:45,061 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-21 23:48:45,062 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:45,596 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-21 23:48:45,597 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:46,913 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-21 23:48:46,914 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:48,256 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-21 23:48:48,256 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:48,643 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-21 23:48:48,644 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:49,037 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-21 23:48:49,037 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:49,463 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-21 23:48:49,463 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:49,838 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 23:48:49,838 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:50,244 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-21 23:48:50,244 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:50,647 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:48:50,648 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:51,051 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-21 23:48:51,051 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:51,452 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 23:48:51,452 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:51,851 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-21 23:48:51,851 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:52,232 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 23:48:52,233 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:52,634 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-21 23:48:52,634 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:53,020 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:48:53,020 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:53,471 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-21 23:48:53,472 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:48:53,874 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 23:48:53,874 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-21 23:48:54,119 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:48:54,188 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:48:54,189 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 23:48:54,190 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-21 23:48:54,574 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:48:54,642 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-21 23:48:54,643 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 23:48:54,643 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 23:48:54,644 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 23:48:54,645 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-21 23:48:54,902 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 23:48:54,981 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 23:48:55,023 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 23:48:55,111 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-21 23:48:55,112 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:48:55,113 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:48:55,113 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:48:55,114 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:48:55,419 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 23:48:55,493 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 23:48:55,536 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 23:48:55,539 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-21 23:48:55,540 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 23:48:57,923 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-21 23:48:57,924 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:48:58,540 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 23:48:58,541 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:48:58,542 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:48:58,544 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:48:58,543 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:48:58,832 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 23:48:58,847 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 23:48:58,924 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 23:48:59,021 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-21 23:48:59,022 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 23:49:01,406 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-21 23:49:01,407 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:49:01,928 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 23:49:01,929 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:49:01,930 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:49:01,932 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:49:01,935 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:49:02,262 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 23:49:02,281 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 23:49:02,385 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 23:49:02,454 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-21 23:49:02,455 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-21 23:49:05,835 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-21 23:49:05,835 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:49:06,361 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 23:49:06,361 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-21 23:49:06,765 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-MDT0000
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-OST0000

2017-08-21 23:49:06,765 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-MDT0000
2017-08-21 23:49:07,089 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-MDT0000) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-MDT0000 exit_status 0 output:
 
2017-08-21 23:49:07,090 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-OST0000
2017-08-21 23:49:07,539 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-OST0000) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/testfs-OST0000 exit_status 0 output:
 
2017-08-21 23:49:07,540 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:49:07,983 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 23:49:07,983 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:49:08,391 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 23:49:08,392 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:49:08,727 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 23:49:08,728 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 23:49:09,245 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 23:49:09,246 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-21 23:49:09,917 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-21 23:49:09,918 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 23:49:10,285 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 23:49:10,286 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-21 23:49:10,862 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-21 23:49:10,863 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-21 23:49:11,226 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-21 23:49:11,226 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-21 23:49:11,939 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-21 23:49:11,939 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-21 23:49:12,358 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-21 23:49:12,358 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-21 23:49:12,665 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-21 23:49:12,665 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-21 23:49:12,969 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-21 23:49:12,969 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 23:49:12,970 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:49:13,335 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-21 23:49:13,336 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-21 23:49:15,865 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-21 23:49:17,039 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T06:49:15.304637', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 23:49:25,053 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T06:49:15.304637', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-21 23:49:25,123 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T06:49:15.398655', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 23:49:25,148 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T06:49:15.398655', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-21 23:49:25,219 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T06:49:15.454774', u'message': u'Validation host configuration lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 23:49:25,248 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T06:49:15.454774', u'message': u'Validation host configuration lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-21 23:49:25,322 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T06:49:15.535824', u'message': u'Validation host configuration lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 23:49:25,355 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T06:49:15.535824', u'message': u'Validation host configuration lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-21 23:49:29,013 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/5/'], u'complete': False, u'created_at': u'2017-08-22T06:49:25.316045', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 23:49:34,247 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/5/'], u'complete': True, u'created_at': u'2017-08-22T06:49:25.316045', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-21 23:49:34,278 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/6/'], u'complete': True, u'created_at': u'2017-08-22T06:49:26.106065', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u''}
2017-08-21 23:49:34,310 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/6/'], u'complete': True, u'created_at': u'2017-08-22T06:49:26.106065', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u''}
2017-08-21 23:49:34,341 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/7/'], u'complete': False, u'created_at': u'2017-08-22T06:49:26.757547', u'message': u'Deploying agent to host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u''}
2017-08-21 23:49:35,409 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/7/'], u'complete': True, u'created_at': u'2017-08-22T06:49:26.757547', u'message': u'Deploying agent to host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u''}
2017-08-21 23:49:35,442 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/8/'], u'complete': False, u'created_at': u'2017-08-22T06:49:27.416717', u'message': u'Deploying agent to host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-21 23:49:36,514 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/8/'], u'complete': True, u'created_at': u'2017-08-22T06:49:27.416717', u'message': u'Deploying agent to host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-21 23:49:37,458 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:38,532 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:39,598 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:40,668 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:41,746 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:42,819 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:43,888 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:44,947 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:46,013 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:47,066 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:48,127 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:49,200 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:50,272 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:51,336 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:52,400 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:53,466 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:54,519 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:55,590 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:56,663 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:57,749 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:58,814 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:49:59,878 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:00,954 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:02,021 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:03,092 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:04,158 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:05,230 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:06,294 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:07,339 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:08,410 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:09,484 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:10,552 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:11,626 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:12,692 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:13,764 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:14,987 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:16,062 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:17,113 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:18,183 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:19,249 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:20,304 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:21,375 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:22,445 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:23,518 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:24,577 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:25,644 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:26,726 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:27,801 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:28,862 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:29,937 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:31,011 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:32,080 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:33,152 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:34,219 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:35,285 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:36,334 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:37,380 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:38,447 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:39,517 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:40,574 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:41,638 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:42,712 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:43,784 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:44,855 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:45,930 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:46,986 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:48,047 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:49,115 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:50,181 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:51,235 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:52,294 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-21 23:50:54,664 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/'], u'complete': False, u'created_at': u'2017-08-22T06:50:52.889509', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u''}
2017-08-22 00:00:17,234 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/', u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/'], u'complete': True, u'created_at': u'2017-08-22T06:50:52.889509', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:00:17,312 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/', u'/api/job/28/', u'/api/job/29/', u'/api/job/30/'], u'complete': False, u'created_at': u'2017-08-22T06:50:53.148917', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:01:39,608 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/', u'/api/job/27/', u'/api/job/28/', u'/api/job/29/', u'/api/job/30/'], u'complete': True, u'created_at': u'2017-08-22T06:50:53.148917', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:01:39,690 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/'], u'complete': False, u'created_at': u'2017-08-22T06:50:53.397228', u'message': u'Setup managed host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:01:41,953 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/31/', u'/api/job/32/', u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/', u'/api/job/38/', u'/api/job/39/', u'/api/job/40/', u'/api/job/41/'], u'complete': True, u'created_at': u'2017-08-22T06:50:53.397228', u'message': u'Setup managed host lotus-59vm17.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:01:42,016 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/', u'/api/job/47/', u'/api/job/48/', u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/'], u'complete': False, u'created_at': u'2017-08-22T06:50:53.595720', u'message': u'Setup managed host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:01:57,423 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/42/', u'/api/job/43/', u'/api/job/44/', u'/api/job/45/', u'/api/job/46/', u'/api/job/47/', u'/api/job/48/', u'/api/job/49/', u'/api/job/50/', u'/api/job/51/', u'/api/job/52/'], u'complete': True, u'created_at': u'2017-08-22T06:50:53.595720', u'message': u'Setup managed host lotus-59vm18.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:01:58,683 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:01:58,684 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:01:59,088 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:01:59,088 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:01:59,545 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:01:59,545 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:01:59,942 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:01:59,943 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:02:01,583 - test - INFO - Configuring power control on [u'lotus-59vm15.lotus.hpdd.lab.intel.com', u'lotus-59vm16.lotus.hpdd.lab.intel.com', u'lotus-59vm17.lotus.hpdd.lab.intel.com', u'lotus-59vm18.lotus.hpdd.lab.intel.com']
2017-08-22 00:02:02,031 - test - DEBUG - Created /api/power_control_type/14/
2017-08-22 00:02:02,124 - test - DEBUG - Created /api/power_control_device/1/
2017-08-22 00:02:03,139 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/1/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm15', u'id': 1, u'resource_uri': u'/api/power_control_device_outlet/1/'}
2017-08-22 00:02:04,161 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/2/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm16', u'id': 2, u'resource_uri': u'/api/power_control_device_outlet/2/'}
2017-08-22 00:02:05,255 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/3/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm17', u'id': 3, u'resource_uri': u'/api/power_control_device_outlet/3/'}
2017-08-22 00:02:06,341 - test - DEBUG - Created {u'has_power': None, u'host': u'/api/host/4/', u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm18', u'id': 4, u'resource_uri': u'/api/power_control_device_outlet/4/'}
2017-08-22 00:02:06,446 - test - DEBUG - Created {u'has_power': None, u'host': None, u'device': u'/api/power_control_device/1/', u'identifier': u'lotus-59vm19', u'id': 5, u'resource_uri': u'/api/power_control_device_outlet/5/'}
2017-08-22 00:02:06,446 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-22 00:02:06,447 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:02:09,077 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: []
2017-08-22 00:02:09,078 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-22 00:02:09,078 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:02:11,745 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com']
2017-08-22 00:02:11,745 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-22 00:02:11,746 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:02:14,339 - test - DEBUG - Fencible nodes on lotus-59vm17.lotus.hpdd.lab.intel.com are: ['lotus-59vm17.lotus.hpdd.lab.intel.com']
2017-08-22 00:02:14,340 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-22 00:02:14,340 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:02:17,006 - test - DEBUG - Fencible nodes on lotus-59vm18.lotus.hpdd.lab.intel.com are: ['lotus-59vm18.lotus.hpdd.lab.intel.com', 'lotus-59vm17.lotus.hpdd.lab.intel.com']
2017-08-22 00:02:18,007 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-22 00:02:18,008 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:02:20,667 - test - DEBUG - Fencible nodes on lotus-59vm15.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-22 00:02:20,667 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-22 00:02:20,668 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:02:23,282 - test - DEBUG - Fencible nodes on lotus-59vm16.lotus.hpdd.lab.intel.com are: ['lotus-59vm15.lotus.hpdd.lab.intel.com', 'lotus-59vm16.lotus.hpdd.lab.intel.com']
2017-08-22 00:02:23,283 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-22 00:02:23,284 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:02:25,899 - test - DEBUG - Fencible nodes on lotus-59vm17.lotus.hpdd.lab.intel.com are: ['lotus-59vm18.lotus.hpdd.lab.intel.com', 'lotus-59vm17.lotus.hpdd.lab.intel.com']
2017-08-22 00:02:25,899 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: fence_chroma -o list
2017-08-22 00:02:25,899 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:02:28,562 - test - DEBUG - Fencible nodes on lotus-59vm18.lotus.hpdd.lab.intel.com are: ['lotus-59vm18.lotus.hpdd.lab.intel.com', 'lotus-59vm17.lotus.hpdd.lab.intel.com']
2017-08-22 00:02:30,282 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/47/', u'volume_nodes': [{u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 4, u'resource_uri': u'/api/volume_node/4/', u'id': 4, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 3, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/14/', u'id': 14, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/19/', u'id': 19, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/21/', u'volume_nodes': [{u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 4, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 3, u'resource_uri': u'/api/volume_node/6/', u'id': 6, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/11/', u'id': 11, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/16/', u'id': 16, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/49/', u'volume_nodes': [{u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 4, u'resource_uri': u'/api/volume_node/5/', u'id': 5, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/15/', u'id': 15, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/21/', u'id': 21, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 3, u'resource_uri': u'/api/volume_node/22/', u'id': 22, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/45/', u'volume_nodes': [{u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 4, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 3, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/13/', u'id': 13, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/18/', u'id': 18, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/24/', u'volume_nodes': [{u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 4, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 3, u'resource_uri': u'/api/volume_node/7/', u'id': 7, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/12/', u'id': 12, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/17/', u'id': 17, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/5/', u'id': 5, u'size': u'10737418240'}]'
2017-08-22 00:02:30,283 - test - DEBUG -         self.wait_until_true(lambda: len(self.get_shared_volumes(required_hosts)) >= expected_volumes)
 evaluated to True
2017-08-22 00:02:32,077 - test - INFO - Found these HA volumes: '[{u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/47/', u'volume_nodes': [{u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 4, u'resource_uri': u'/api/volume_node/4/', u'id': 4, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 1, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 3, u'resource_uri': u'/api/volume_node/9/', u'id': 9, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 2, u'resource_uri': u'/api/volume_node/14/', u'id': 14, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 1, u'primary': True, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'host_id': 1, u'resource_uri': u'/api/volume_node/19/', u'id': 19, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/1/', u'id': 1, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/21/', u'volume_nodes': [{u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 4, u'resource_uri': u'/api/volume_node/1/', u'id': 1, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 2, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 3, u'resource_uri': u'/api/volume_node/6/', u'id': 6, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': True, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 2, u'resource_uri': u'/api/volume_node/11/', u'id': 11, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 2, u'primary': False, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14', u'host_id': 1, u'resource_uri': u'/api/volume_node/16/', u'id': 16, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk14', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/2/', u'id': 2, u'size': u'10737418240'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/49/', u'volume_nodes': [{u'use': True, u'volume_id': 3, u'primary': True, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 4, u'resource_uri': u'/api/volume_node/5/', u'id': 5, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 2, u'resource_uri': u'/api/volume_node/15/', u'id': 15, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 3, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 1, u'resource_uri': u'/api/volume_node/21/', u'id': 21, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 3, u'primary': False, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'host_id': 3, u'resource_uri': u'/api/volume_node/22/', u'id': 22, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/3/', u'id': 3, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'ZfsPool', u'storage_resource': u'/api/storage_resource/45/', u'volume_nodes': [{u'use': True, u'volume_id': 4, u'primary': False, u'host': u'/api/host/4/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 4, u'resource_uri': u'/api/volume_node/3/', u'id': 3, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 4, u'primary': True, u'host': u'/api/host/3/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 3, u'resource_uri': u'/api/volume_node/8/', u'id': 8, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/2/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 2, u'resource_uri': u'/api/volume_node/13/', u'id': 13, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 4, u'primary': False, u'host': u'/api/host/1/', u'path': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'host_id': 1, u'resource_uri': u'/api/volume_node/18/', u'id': 18, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/4/', u'id': 4, u'size': u'10672993730'}, {u'status': u'configured-ha', u'kind': u'SCSI device', u'storage_resource': u'/api/storage_resource/24/', u'volume_nodes': [{u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/4/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 4, u'resource_uri': u'/api/volume_node/2/', u'id': 2, u'host_label': u'lotus-59vm18.lotus.hpdd.lab.intel.com'}, {u'use': False, u'volume_id': 5, u'primary': False, u'host': u'/api/host/3/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 3, u'resource_uri': u'/api/volume_node/7/', u'id': 7, u'host_label': u'lotus-59vm17.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': False, u'host': u'/api/host/2/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 2, u'resource_uri': u'/api/volume_node/12/', u'id': 12, u'host_label': u'lotus-59vm16.lotus.hpdd.lab.intel.com'}, {u'use': True, u'volume_id': 5, u'primary': True, u'host': u'/api/host/1/', u'path': u'/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12', u'host_id': 1, u'resource_uri': u'/api/volume_node/17/', u'id': 17, u'host_label': u'lotus-59vm15.lotus.hpdd.lab.intel.com'}], u'filesystem_type': None, u'label': u'disk12', u'usable_for_lustre': True, u'resource_uri': u'/api/volume/5/', u'id': 5, u'size': u'10737418240'}]'
2017-08-22 00:02:33,671 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/59/', u'/api/job/60/', u'/api/job/61/', u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/', u'/api/job/70/', u'/api/job/71/', u'/api/job/72/', u'/api/job/73/', u'/api/job/74/', u'/api/job/75/'], u'complete': False, u'created_at': u'2017-08-22T07:02:32.014676', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/19/', u'id': 19, u'logs': u'\n\n\n'}
2017-08-22 00:06:30,411 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/59/', u'/api/job/60/', u'/api/job/61/', u'/api/job/62/', u'/api/job/63/', u'/api/job/64/', u'/api/job/65/', u'/api/job/66/', u'/api/job/67/', u'/api/job/68/', u'/api/job/69/', u'/api/job/70/', u'/api/job/71/', u'/api/job/72/', u'/api/job/73/', u'/api/job/74/', u'/api/job/75/'], u'complete': True, u'created_at': u'2017-08-22T07:02:32.014676', u'message': u'Creating filesystem testfs', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/19/', u'id': 19, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:06:30,412 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-22 00:06:30,412 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:30,874 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-22 00:06:30,874 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:31,339 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-22 00:06:31,339 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:31,797 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cibadmin -t 1 --query
2017-08-22 00:06:31,797 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:37,709 - test - DEBUG - MGS: should be running on lotus-59vm15.lotus.hpdd.lab.intel.com (actual: lotus-59vm15.lotus.hpdd.lab.intel.com)
2017-08-22 00:06:37,709 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: crm_resource -r MGS_1debe8 -W
2017-08-22 00:06:37,709 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:38,141 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-MGS_1debe8 -N lotus-59vm15.lotus.hpdd.lab.intel.com -G -d 0
2017-08-22 00:06:38,141 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:38,594 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-22 00:06:38,594 - test - DEBUG - testfs-MDT0000: should be running on lotus-59vm16.lotus.hpdd.lab.intel.com (actual: lotus-59vm16.lotus.hpdd.lab.intel.com)
2017-08-22 00:06:38,594 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: crm_resource -r testfs-MDT0000_4954b6 -W
2017-08-22 00:06:38,595 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:39,042 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-testfs-MDT0000_4954b6 -N lotus-59vm16.lotus.hpdd.lab.intel.com -G -d 0
2017-08-22 00:06:39,043 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:39,446 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-22 00:06:39,446 - test - DEBUG - testfs-OST0000: should be running on lotus-59vm17.lotus.hpdd.lab.intel.com (actual: lotus-59vm17.lotus.hpdd.lab.intel.com)
2017-08-22 00:06:39,446 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: crm_resource -r testfs-OST0000_18abb0 -W
2017-08-22 00:06:39,447 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:39,899 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-testfs-OST0000_18abb0 -N lotus-59vm17.lotus.hpdd.lab.intel.com -G -d 0
2017-08-22 00:06:39,899 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:40,340 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-22 00:06:40,340 - test - DEBUG - testfs-OST0001: should be running on lotus-59vm18.lotus.hpdd.lab.intel.com (actual: lotus-59vm18.lotus.hpdd.lab.intel.com)
2017-08-22 00:06:40,341 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: crm_resource -r testfs-OST0001_9c8080 -W
2017-08-22 00:06:40,341 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:40,787 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: crm_attribute -t status -n fail-count-testfs-OST0001_9c8080 -N lotus-59vm18.lotus.hpdd.lab.intel.com -G -d 0
2017-08-22 00:06:40,787 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:41,190 - test - DEBUG - Manager says it's OK, pacemaker says: True
2017-08-22 00:06:41,953 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:06:41,953 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:42,353 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_corosync_reverse_dependencies (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:06:42,353 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:42,730 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:06:42,730 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:43,131 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_corosync_reverse_dependencies (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:06:43,131 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:44,044 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 00:06:44,044 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:06:44,044 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:44,434 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-22 00:06:44,435 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_corosync_state_change_1 (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:06:44,435 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:44,819 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 00:06:44,819 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:06:44,819 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:45,220 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-22 00:06:45,220 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_corosync_state_change_1 (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:06:45,220 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:45,603 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:06:45,604 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:46,199 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 00:06:46,199 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:46,902 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-22 00:06:46,903 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:47,703 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:06:47,704 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:48,099 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 00:06:48,099 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:48,464 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 00:06:48,464 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:48,837 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:06:48,838 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:49,210 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 00:06:49,210 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:49,613 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 00:06:49,613 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:06:49,996 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-22 00:07:19,338 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-22 00:07:20,938 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-22 00:07:22,715 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-22 00:08:38,762 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-22 00:08:40,812 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-22 00:08:41,427 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-22 00:08:41,427 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-22 00:08:41,428 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-22 00:08:45,478 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:08:45,478 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:45,874 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:08:45,874 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:47,430 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:08:47,431 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:47,834 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:08:47,835 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:48,376 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:08:48,376 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:48,777 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:08:48,777 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:49,463 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:08:49,463 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:49,864 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:08:49,864 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:50,399 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:08:50,399 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:50,795 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:08:50,796 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:51,170 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:08:51,171 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:51,556 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:08:51,556 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:51,948 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 00:08:51,949 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:08:52,350 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 00:08:52,350 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:21,742 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 00:09:21,743 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:23,009 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 00:09:23,009 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:24,318 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:09:24,318 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:24,696 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 00:09:24,696 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:25,090 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 00:09:25,090 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:25,534 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 00:09:25,535 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:25,909 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 00:09:25,909 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:26,311 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 00:09:26,312 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:53,697 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 00:09:53,698 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:54,924 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 00:09:54,925 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:56,197 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:09:56,197 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:56,599 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 00:09:56,599 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:56,998 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 00:09:56,999 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:57,423 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 00:09:57,423 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:57,806 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 00:09:57,807 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:58,200 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 00:09:58,200 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:58,598 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:09:58,599 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:59,051 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:09:59,051 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:59,446 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 00:09:59,447 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:09:59,852 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 00:09:59,852 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:10:00,227 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 00:10:00,228 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:10:00,613 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 00:10:00,613 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:10:00,990 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:10:00,990 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:10:01,391 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:10:01,391 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:10:01,794 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 00:10:01,795 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 00:10:02,106 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:10:02,225 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:10:02,226 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 00:10:02,227 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 00:10:02,666 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:10:02,672 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:10:02,673 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:10:02,673 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:10:02,674 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:10:02,675 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:10:02,997 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:10:03,057 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:10:03,108 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:10:03,109 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:10:03,110 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:10:03,111 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:10:03,112 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:10:03,113 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:10:03,534 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:10:03,650 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:10:03,655 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:10:03,666 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:10:03,667 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-22 00:10:05,030 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-22 00:10:05,031 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:10:05,569 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-22 00:10:05,570 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:10:05,571 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:10:05,572 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:10:05,573 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:10:05,891 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:10:05,896 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:10:06,072 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:10:06,085 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:10:06,085 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-22 00:10:08,592 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-22 00:10:08,593 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:10:09,233 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-22 00:10:09,234 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:10:09,235 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:10:09,237 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:10:09,237 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:10:09,638 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:10:09,642 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:10:09,677 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:10:09,686 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:10:09,686 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-22 00:10:13,068 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-22 00:10:13,068 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:10:13,581 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-22 00:10:13,582 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-22 00:10:13,873 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0001
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS

2017-08-22 00:10:13,873 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0001
2017-08-22 00:10:14,179 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0001) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11/testfs-OST0001 exit_status 0 output:
 
2017-08-22 00:10:14,179 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000
2017-08-22 00:10:14,598 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/testfs-OST0000 exit_status 0 output:
 
2017-08-22 00:10:14,599 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS
2017-08-22 00:10:15,027 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS) command zfs destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15/MGS exit_status 0 output:
 
2017-08-22 00:10:15,027 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:10:15,453 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-22 00:10:15,453 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:10:15,765 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-22 00:10:15,765 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:10:16,172 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-22 00:10:16,173 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-22 00:10:16,620 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-22 00:10:16,621 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-22 00:10:17,274 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-22 00:10:17,275 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-22 00:10:17,743 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-22 00:10:17,743 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-22 00:10:18,289 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-22 00:10:18,290 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-22 00:10:18,763 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-22 00:10:18,763 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-22 00:10:19,405 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-22 00:10:19,405 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:10:19,822 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-22 00:10:19,822 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:10:20,105 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-22 00:10:20,105 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:10:20,519 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-22 00:10:20,519 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-22 00:10:20,519 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:10:20,892 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-22 00:10:20,892 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:10:22,810 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-22 00:10:24,439 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T07:10:22.897030', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-22 00:10:31,791 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T07:10:22.897030', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-22 00:10:32,562 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T07:10:22.985835', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-22 00:10:32,598 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T07:10:22.985835', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-22 00:10:34,518 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/'], u'complete': False, u'created_at': u'2017-08-22T07:10:32.541607', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-22 00:10:40,798 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T07:10:32.541607', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-22 00:10:40,825 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/4/'], u'complete': False, u'created_at': u'2017-08-22T07:10:33.291362', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-22 00:10:41,893 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T07:10:33.291362', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-22 00:10:42,488 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:43,541 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:44,596 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:45,654 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:46,702 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:47,751 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:48,807 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:49,858 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:50,912 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:51,964 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:53,014 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:54,067 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:55,114 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:56,160 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:57,203 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:58,257 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:10:59,313 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:00,363 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:01,410 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:02,459 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:03,505 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:04,554 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:05,606 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:06,660 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:07,709 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:08,758 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:09,809 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:10,856 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:11,907 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:12,958 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:14,008 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:15,061 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:16,113 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:17,165 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:18,217 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:19,271 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:20,322 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:21,389 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:22,440 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:23,493 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:24,543 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:25,595 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:26,648 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:27,699 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:28,754 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:29,810 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:30,858 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:31,907 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:32,960 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:33,997 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:35,050 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:36,098 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:37,155 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:38,207 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:39,262 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:40,305 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:41,350 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:42,395 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:43,441 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:44,489 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:45,538 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:46,578 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:47,626 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:48,687 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:49,738 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:50,788 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:51,835 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:52,881 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:53,921 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:54,974 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:56,021 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:57,069 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:58,116 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:11:59,159 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:12:00,193 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:12:01,243 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:12:02,295 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:12:04,094 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': False, u'created_at': u'2017-08-22T07:12:02.797127', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-22 00:22:18,536 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': True, u'created_at': u'2017-08-22T07:12:02.797127', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:22:18,623 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': True, u'created_at': u'2017-08-22T07:12:03.108422', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:22:18,696 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': True, u'created_at': u'2017-08-22T07:12:03.108422', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:22:19,339 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:22:19,339 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:22:19,817 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:22:19,817 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:22:20,840 - test - DEBUG - set_state /api/corosync_configuration/1/ started
2017-08-22 00:22:20,917 - test - WARNING - set_state /api/corosync_configuration/1/ started - no-op
2017-08-22 00:22:20,917 - test - DEBUG - assertValue state /api/corosync_configuration/1/ started
2017-08-22 00:22:21,000 - test - DEBUG - set_state /api/corosync_configuration/1/ stopped
2017-08-22 00:22:21,197 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/27/', u'/api/job/28/'], u'complete': False, u'created_at': u'2017-08-22T07:22:20.325698', u'message': u'Stop Corosync on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u''}
2017-08-22 00:22:36,908 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/27/', u'/api/job/28/'], u'complete': True, u'created_at': u'2017-08-22T07:22:20.325698', u'message': u'Stop Corosync on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/7/', u'id': 7, u'logs': u'\n'}
2017-08-22 00:22:36,909 - test - DEBUG - assertValue state /api/corosync_configuration/1/ stopped
2017-08-22 00:22:37,015 - test - DEBUG - assertValue state /api/pacemaker_configuration/1/ stopped
2017-08-22 00:22:37,065 - test - DEBUG - set_state /api/corosync_configuration/1/ started
2017-08-22 00:22:37,205 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/29/'], u'complete': False, u'created_at': u'2017-08-22T07:22:36.384004', u'message': u'Start Corosync on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-22 00:22:46,592 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/29/'], u'complete': True, u'created_at': u'2017-08-22T07:22:36.384004', u'message': u'Start Corosync on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-22 00:22:46,593 - test - DEBUG - assertValue state /api/corosync_configuration/1/ started
2017-08-22 00:22:46,684 - test - DEBUG - assertValue state /api/pacemaker_configuration/1/ stopped
2017-08-22 00:22:46,749 - test - DEBUG - set_state /api/corosync_configuration/1/ stopped
2017-08-22 00:22:46,928 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/30/'], u'complete': False, u'created_at': u'2017-08-22T07:22:46.091097', u'message': u'Stop Corosync on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u''}
2017-08-22 00:22:56,319 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/30/'], u'complete': True, u'created_at': u'2017-08-22T07:22:46.091097', u'message': u'Stop Corosync on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u''}
2017-08-22 00:22:56,320 - test - DEBUG - assertValue state /api/corosync_configuration/1/ stopped
2017-08-22 00:22:56,392 - test - DEBUG - set_state /api/pacemaker_configuration/1/ started
2017-08-22 00:22:56,570 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/31/', u'/api/job/32/'], u'complete': False, u'created_at': u'2017-08-22T07:22:55.726520', u'message': u'Start Pacemaker on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u''}
2017-08-22 00:23:13,303 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/31/', u'/api/job/32/'], u'complete': True, u'created_at': u'2017-08-22T07:22:55.726520', u'message': u'Start Pacemaker on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/10/', u'id': 10, u'logs': u'\n'}
2017-08-22 00:23:13,304 - test - DEBUG - assertValue state /api/pacemaker_configuration/1/ started
2017-08-22 00:23:13,376 - test - DEBUG - assertValue state /api/corosync_configuration/1/ started
2017-08-22 00:23:13,434 - test - DEBUG - set_state /api/pacemaker_configuration/1/ stopped
2017-08-22 00:23:13,597 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/33/'], u'complete': False, u'created_at': u'2017-08-22T07:23:12.762352', u'message': u'Stop Pacemaker on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u''}
2017-08-22 00:23:20,912 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/33/'], u'complete': True, u'created_at': u'2017-08-22T07:23:12.762352', u'message': u'Stop Pacemaker on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/11/', u'id': 11, u'logs': u''}
2017-08-22 00:23:20,912 - test - DEBUG - assertValue state /api/pacemaker_configuration/1/ stopped
2017-08-22 00:23:20,981 - test - DEBUG - assertValue state /api/corosync_configuration/1/ started
2017-08-22 00:23:21,049 - test - DEBUG - set_state /api/pacemaker_configuration/1/ started
2017-08-22 00:23:21,245 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/34/'], u'complete': False, u'created_at': u'2017-08-22T07:23:20.394786', u'message': u'Start Pacemaker on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u''}
2017-08-22 00:23:28,572 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/34/'], u'complete': True, u'created_at': u'2017-08-22T07:23:20.394786', u'message': u'Start Pacemaker on lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/12/', u'id': 12, u'logs': u''}
2017-08-22 00:23:28,573 - test - DEBUG - assertValue state /api/pacemaker_configuration/1/ started
2017-08-22 00:23:28,627 - test - DEBUG - set_state /api/corosync_configuration/2/ started
2017-08-22 00:23:28,693 - test - WARNING - set_state /api/corosync_configuration/2/ started - no-op
2017-08-22 00:23:28,693 - test - DEBUG - assertValue state /api/corosync_configuration/2/ started
2017-08-22 00:23:28,786 - test - DEBUG - set_state /api/corosync_configuration/2/ stopped
2017-08-22 00:23:28,942 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/35/', u'/api/job/36/'], u'complete': False, u'created_at': u'2017-08-22T07:23:28.104975', u'message': u'Stop Corosync on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/13/', u'id': 13, u'logs': u''}
2017-08-22 00:23:46,743 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/35/', u'/api/job/36/'], u'complete': True, u'created_at': u'2017-08-22T07:23:28.104975', u'message': u'Stop Corosync on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/13/', u'id': 13, u'logs': u'\n'}
2017-08-22 00:23:46,743 - test - DEBUG - assertValue state /api/corosync_configuration/2/ stopped
2017-08-22 00:23:46,868 - test - DEBUG - assertValue state /api/pacemaker_configuration/2/ stopped
2017-08-22 00:23:46,943 - test - DEBUG - set_state /api/corosync_configuration/2/ started
2017-08-22 00:23:47,151 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/37/'], u'complete': False, u'created_at': u'2017-08-22T07:23:46.299414', u'message': u'Start Corosync on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/14/', u'id': 14, u'logs': u''}
2017-08-22 00:23:56,536 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/37/'], u'complete': True, u'created_at': u'2017-08-22T07:23:46.299414', u'message': u'Start Corosync on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/14/', u'id': 14, u'logs': u''}
2017-08-22 00:23:56,537 - test - DEBUG - assertValue state /api/corosync_configuration/2/ started
2017-08-22 00:23:56,617 - test - DEBUG - assertValue state /api/pacemaker_configuration/2/ stopped
2017-08-22 00:23:56,663 - test - DEBUG - set_state /api/corosync_configuration/2/ stopped
2017-08-22 00:23:56,807 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/38/'], u'complete': False, u'created_at': u'2017-08-22T07:23:55.979049', u'message': u'Stop Corosync on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/15/', u'id': 15, u'logs': u''}
2017-08-22 00:24:05,142 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/38/'], u'complete': True, u'created_at': u'2017-08-22T07:23:55.979049', u'message': u'Stop Corosync on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/15/', u'id': 15, u'logs': u''}
2017-08-22 00:24:05,143 - test - DEBUG - assertValue state /api/corosync_configuration/2/ stopped
2017-08-22 00:24:05,202 - test - DEBUG - set_state /api/pacemaker_configuration/2/ started
2017-08-22 00:24:05,379 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/39/', u'/api/job/40/'], u'complete': False, u'created_at': u'2017-08-22T07:24:04.531457', u'message': u'Start Pacemaker on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/16/', u'id': 16, u'logs': u''}
2017-08-22 00:24:22,121 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/39/', u'/api/job/40/'], u'complete': True, u'created_at': u'2017-08-22T07:24:04.531457', u'message': u'Start Pacemaker on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/16/', u'id': 16, u'logs': u'\n'}
2017-08-22 00:24:22,121 - test - DEBUG - assertValue state /api/pacemaker_configuration/2/ started
2017-08-22 00:24:22,197 - test - DEBUG - assertValue state /api/corosync_configuration/2/ started
2017-08-22 00:24:22,265 - test - DEBUG - set_state /api/pacemaker_configuration/2/ stopped
2017-08-22 00:24:22,439 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/41/'], u'complete': False, u'created_at': u'2017-08-22T07:24:21.601249', u'message': u'Stop Pacemaker on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/17/', u'id': 17, u'logs': u''}
2017-08-22 00:24:30,781 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/41/'], u'complete': True, u'created_at': u'2017-08-22T07:24:21.601249', u'message': u'Stop Pacemaker on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/17/', u'id': 17, u'logs': u''}
2017-08-22 00:24:30,781 - test - DEBUG - assertValue state /api/pacemaker_configuration/2/ stopped
2017-08-22 00:24:30,852 - test - DEBUG - assertValue state /api/corosync_configuration/2/ started
2017-08-22 00:24:30,918 - test - DEBUG - set_state /api/pacemaker_configuration/2/ started
2017-08-22 00:24:31,079 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/42/'], u'complete': False, u'created_at': u'2017-08-22T07:24:30.247018', u'message': u'Start Pacemaker on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/18/', u'id': 18, u'logs': u''}
2017-08-22 00:24:39,443 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/42/'], u'complete': True, u'created_at': u'2017-08-22T07:24:30.247018', u'message': u'Start Pacemaker on lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/18/', u'id': 18, u'logs': u''}
2017-08-22 00:24:39,444 - test - DEBUG - assertValue state /api/pacemaker_configuration/2/ started
2017-08-22 00:24:39,509 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:24:39,509 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:39,914 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_corosync_state_change_1 (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:24:39,914 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:40,313 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:24:40,314 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:40,726 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_corosync_state_change_1 (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:24:40,726 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:41,355 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 00:24:41,355 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:24:41,355 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:41,754 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-22 00:24:41,754 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_host_goes_down (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:24:41,754 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:42,141 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 00:24:42,141 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:24:42,142 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:42,534 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-22 00:24:42,534 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_host_goes_down (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:24:42,534 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:42,928 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:24:42,928 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:43,648 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 00:24:43,649 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:44,345 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-22 00:24:44,346 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:45,038 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:24:45,039 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:45,437 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 00:24:45,437 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:45,812 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 00:24:45,812 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:46,197 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:24:46,197 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:46,571 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 00:24:46,572 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:46,948 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 00:24:46,948 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:24:47,323 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-22 00:25:12,594 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-22 00:25:14,275 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-22 00:25:16,087 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-22 00:26:32,891 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-22 00:26:34,804 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-22 00:26:35,379 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-22 00:26:35,379 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-22 00:26:35,380 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-22 00:26:39,379 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:26:39,379 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:39,836 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:26:39,837 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:41,392 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:26:41,392 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:41,789 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:26:41,789 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:42,325 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:26:42,326 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:42,722 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:26:42,722 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:43,097 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:26:43,097 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:43,492 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:26:43,493 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:43,894 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:26:43,894 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:44,297 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:26:44,297 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:44,670 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:26:44,670 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:45,044 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:26:45,045 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:45,419 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 00:26:45,419 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:45,819 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 00:26:45,819 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:48,662 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 00:26:48,662 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:49,925 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 00:26:49,926 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:51,131 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:26:51,131 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:51,518 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 00:26:51,519 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:51,887 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 00:26:51,888 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:52,321 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 00:26:52,321 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:52,698 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 00:26:52,699 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:53,094 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 00:26:53,094 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:54,751 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 00:26:54,751 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:55,909 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 00:26:55,909 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:57,145 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:26:57,145 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:57,543 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 00:26:57,544 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:57,992 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 00:26:57,992 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:58,442 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 00:26:58,442 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:58,846 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 00:26:58,846 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:59,246 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 00:26:59,246 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:26:59,629 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:26:59,630 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:00,025 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:27:00,025 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:00,407 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 00:27:00,407 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:00,800 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 00:27:00,800 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:01,182 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 00:27:01,182 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:01,586 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 00:27:01,586 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:01,984 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:27:01,985 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:02,379 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:27:02,380 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:02,777 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 00:27:02,778 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 00:27:03,215 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:27:03,215 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:27:03,216 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 00:27:03,217 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 00:27:03,654 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:27:03,667 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:27:03,668 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:27:03,669 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:27:03,670 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:27:03,671 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:27:03,992 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:27:04,041 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:27:04,057 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:27:04,158 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:27:04,159 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:27:04,160 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:27:04,161 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:27:04,162 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:27:04,581 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:27:04,588 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:27:04,660 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:27:04,667 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:27:04,668 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-22 00:27:06,060 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-22 00:27:06,060 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:27:06,571 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 bs=4k count=1; sync
2017-08-22 00:27:06,855 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clearing disk because zfs import failed zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command dd if=/dev/zero of=zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:27:06,856 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:27:06,857 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:27:06,858 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:27:06,859 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:27:07,264 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:27:07,265 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:27:07,267 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:27:07,339 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:27:07,340 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-22 00:27:08,824 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-22 00:27:08,824 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:27:09,432 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 bs=4k count=1; sync
2017-08-22 00:27:09,723 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clearing disk because zfs import failed zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command dd if=/dev/zero of=zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:27:09,724 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:27:09,725 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:27:09,726 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:27:09,726 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:27:10,250 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:27:10,250 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:27:10,256 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:27:10,257 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:27:10,258 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-22 00:27:11,770 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-22 00:27:11,770 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:27:12,400 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-22 00:27:12,401 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-22 00:27:12,689 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-22 00:27:12,689 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:27:12,976 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-22 00:27:12,977 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-22 00:27:13,445 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-22 00:27:13,446 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-22 00:27:14,137 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-22 00:27:14,138 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-22 00:27:14,624 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-22 00:27:14,625 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-22 00:27:15,281 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-22 00:27:15,282 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-22 00:27:15,749 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-22 00:27:15,750 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-22 00:27:16,308 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-22 00:27:16,308 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:27:16,745 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-22 00:27:16,860 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-22 00:27:16,860 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:17,234 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-22 00:27:17,235 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:27:19,226 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-22 00:27:20,252 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T07:27:19.314794', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-22 00:27:28,188 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T07:27:19.314794', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-22 00:27:28,265 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T07:27:19.411898', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-22 00:27:28,296 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T07:27:19.411898', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-22 00:27:30,125 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/'], u'complete': False, u'created_at': u'2017-08-22T07:27:28.111517', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-22 00:27:38,167 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T07:27:28.111517', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-22 00:27:38,188 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/4/'], u'complete': False, u'created_at': u'2017-08-22T07:27:28.863731', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-22 00:27:39,246 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T07:27:28.863731', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-22 00:27:39,868 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:40,920 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:41,974 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:43,034 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:44,085 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:45,146 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:46,195 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:47,241 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:48,290 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:49,340 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:50,392 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:51,443 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:52,496 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:53,548 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:54,599 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:55,651 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:56,703 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:57,755 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:58,808 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:27:59,860 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:00,911 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:01,962 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:03,015 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:04,075 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:05,127 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:06,178 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:07,224 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:08,273 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:09,323 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:10,363 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:11,411 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:12,461 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:13,514 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:14,565 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:15,615 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:16,673 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:17,720 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:18,770 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:19,818 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:20,869 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:21,923 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:22,975 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:24,009 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:25,056 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:28:26,994 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': False, u'created_at': u'2017-08-22T07:28:25.637113', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-22 00:39:16,309 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': True, u'created_at': u'2017-08-22T07:28:25.637113', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:39:16,391 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': False, u'created_at': u'2017-08-22T07:28:25.987985', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:39:18,671 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': True, u'created_at': u'2017-08-22T07:28:25.987985', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:39:19,360 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:39:19,361 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:39:19,815 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:39:19,815 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:39:21,024 - test - DEBUG -         def all_hosts_online():
            corosync_configurations = self.get_list("/api/corosync_configuration/")
            return all([corosync_configuration['corosync_reported_up'] for corosync_configuration in corosync_configurations])
 evaluated to True
2017-08-22 00:39:21,050 - test - DEBUG - remote_command[lotus-59]: [ $(virsh domstate lotus-59vm16) = shut\ off ] || virsh destroy lotus-59vm16
2017-08-22 00:39:21,050 - test - INFO - SSH address = lotus-59, args = {'username': 'root'}
2017-08-22 00:39:22,741 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:39:22,742 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:40:28,890 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:40:28,995 - test - DEBUG -             self.wait_until_true(lambda: self._host_on_off_line(1, False))
 evaluated to True
2017-08-22 00:40:29,070 - test - DEBUG -             self.wait_until_true(lambda: self._host_on_off_line(0, True))
 evaluated to True
2017-08-22 00:40:29,113 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:40:29,113 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:40:32,120 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:40:35,124 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:40:35,124 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:40:38,132 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:40:41,136 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:40:41,136 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:40:44,144 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:40:47,148 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:40:47,148 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:40:50,156 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:40:53,160 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:40:53,160 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:40:56,168 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:40:59,172 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:40:59,172 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:02,181 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:05,185 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:05,185 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:08,192 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:11,196 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:11,196 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:14,204 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:17,208 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:17,208 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:20,218 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:23,222 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:23,222 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:26,230 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:29,234 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:29,234 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:32,244 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:35,248 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:35,248 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:38,256 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:41,260 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:41,260 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:44,268 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:47,272 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:47,278 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:50,288 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:53,292 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:53,292 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:41:56,300 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:41:59,304 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:41:59,305 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:02,312 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:05,316 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:05,316 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:08,324 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:11,328 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:11,328 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:14,336 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:17,340 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:17,340 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:20,348 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:23,352 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:23,352 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:26,360 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:29,364 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:29,364 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:32,372 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:35,376 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:35,376 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:38,384 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:38,384 - test - INFO - attempting to restart lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:38,385 - test - DEBUG - remote_command[lotus-59]: virsh domstate lotus-59vm16
2017-08-22 00:42:38,385 - test - INFO - SSH address = lotus-59, args = {'username': 'root'}
2017-08-22 00:42:39,214 - test - DEBUG - remote_command[lotus-59]: virsh start lotus-59vm16 || [ $(virsh domstate lotus-59vm16) = running ]
2017-08-22 00:42:39,215 - test - INFO - SSH address = lotus-59, args = {'username': 'root'}
2017-08-22 00:42:40,875 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com started successfully
2017-08-22 00:42:43,878 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:43,879 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:46,888 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:49,891 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:49,891 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:49,894 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:52,897 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:52,897 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:55,904 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:42:58,908 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:42:58,908 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:42:59,885 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:43:02,889 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:43:02,889 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:02,892 - test - DEBUG - Unknown socket error when checking lotus-59vm16.lotus.hpdd.lab.intel.com
2017-08-22 00:43:05,895 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:43:05,896 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:06,487 - test - DEBUG -         self.wait_until_true(lambda: self._host_on_off_line(1, True))
 evaluated to False
2017-08-22 00:43:06,563 - test - DEBUG -         self.wait_until_true(lambda: self._host_on_off_line(1, True))
 evaluated to False
2017-08-22 00:43:06,753 - test - DEBUG -         self.wait_until_true(lambda: self._host_on_off_line(1, True))
 evaluated to False
2017-08-22 00:43:07,839 - test - DEBUG -         self.wait_until_true(lambda: self._host_on_off_line(1, True))
 evaluated to False
2017-08-22 00:43:08,932 - test - DEBUG -         self.wait_until_true(lambda: self._host_on_off_line(1, True))
 evaluated to False
2017-08-22 00:43:10,015 - test - DEBUG -         self.wait_until_true(lambda: self._host_on_off_line(1, True))
 evaluated to False
2017-08-22 00:43:11,111 - test - DEBUG -         self.wait_until_true(lambda: self._host_on_off_line(1, True))
 evaluated to False
2017-08-22 00:43:12,198 - test - DEBUG -         self.wait_until_true(lambda: self._host_on_off_line(1, True))
 evaluated to True
2017-08-22 00:43:12,199 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:43:12,199 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:12,655 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_host_goes_down (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:43:12,656 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:12,947 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:43:12,947 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:13,341 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_host_goes_down (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:43:13,341 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:13,987 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 00:43:13,987 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:43:13,987 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:14,385 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-22 00:43:14,385 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_mcast_changes (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:43:14,385 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:14,772 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 00:43:14,772 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:43:14,773 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:15,174 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-22 00:43:15,174 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_mcast_changes (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 00:43:15,175 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:15,557 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:43:15,557 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:16,264 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 00:43:16,265 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:16,964 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-22 00:43:16,965 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:17,660 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:43:17,661 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:18,039 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 00:43:18,040 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:18,414 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 00:43:18,415 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:18,791 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:43:18,791 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:19,183 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 00:43:19,183 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:19,575 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 00:43:19,575 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:43:19,949 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-22 00:43:49,350 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-22 00:43:50,973 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-22 00:43:52,627 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-22 00:45:08,643 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-22 00:45:10,590 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-22 00:45:11,207 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-22 00:45:11,208 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-22 00:45:11,208 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-22 00:45:15,162 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:45:15,162 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:15,607 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:45:15,607 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:16,160 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:45:16,161 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:16,611 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:45:16,611 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:17,138 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:45:17,138 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:17,538 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:45:17,539 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:17,930 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:45:17,930 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:18,327 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:45:18,327 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:18,718 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:45:18,718 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:19,123 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:45:19,123 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:19,496 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 00:45:19,496 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:19,865 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 00:45:19,866 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:20,242 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 00:45:20,243 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:20,649 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 00:45:20,649 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:23,543 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 00:45:23,544 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:24,852 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 00:45:24,853 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:26,146 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:45:26,146 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:26,533 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 00:45:26,533 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:26,909 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 00:45:26,909 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:27,336 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 00:45:27,337 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:27,713 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 00:45:27,713 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:28,117 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 00:45:28,118 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:29,747 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 00:45:29,747 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:30,998 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 00:45:30,999 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:32,303 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 00:45:32,303 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:32,691 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 00:45:32,692 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:33,061 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 00:45:33,062 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:33,514 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 00:45:33,514 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:33,900 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 00:45:33,900 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:34,299 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 00:45:34,299 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:34,694 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:45:34,695 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:35,097 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 00:45:35,097 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:35,472 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 00:45:35,473 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:35,872 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 00:45:35,872 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:36,245 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 00:45:36,246 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:36,620 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 00:45:36,620 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:36,994 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:45:36,995 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:37,447 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:45:37,448 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:37,884 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 00:45:37,885 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 00:45:38,140 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:45:38,320 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:45:38,321 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 00:45:38,322 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 00:45:38,747 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:45:38,751 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:45:38,752 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:45:38,753 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:45:38,753 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:45:38,754 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 00:45:39,160 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:45:39,165 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:45:39,167 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:45:39,282 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 00:45:39,283 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:45:39,284 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:45:39,285 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:45:39,286 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:45:39,609 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:45:39,647 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-22 00:45:39,734 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:45:39,809 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 00:45:39,810 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-22 00:45:42,201 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command partprobe | true exit_status 0 output:
 
2017-08-22 00:45:42,202 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 00:45:42,735 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 bs=4k count=1; sync
2017-08-22 00:45:43,037 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clearing disk because zfs import failed zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command dd if=/dev/zero of=zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:45:43,038 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:45:43,039 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:45:43,039 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:45:43,040 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:45:43,425 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:45:43,447 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 0 output:
 
2017-08-22 00:45:43,455 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:45:43,541 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 exit_status 1 output:
 
2017-08-22 00:45:43,542 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-22 00:45:45,006 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command partprobe | true exit_status 0 output:
 
2017-08-22 00:45:45,006 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13
2017-08-22 00:45:45,497 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 bs=4k count=1; sync
2017-08-22 00:45:45,905 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clearing disk because zfs import failed zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command dd if=/dev/zero of=zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 00:45:45,906 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:45:45,907 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:45:45,909 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:45:45,911 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:45:46,195 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:45:46,300 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:45:46,315 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:45:46,398 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 1 output:
 
2017-08-22 00:45:46,398 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: partprobe | true
2017-08-22 00:45:47,783 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command partprobe | true exit_status 0 output:
 
2017-08-22 00:45:47,784 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:45:48,324 - test - INFO - import zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool import -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-22 00:45:48,324 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zfs list -H -o name
2017-08-22 00:45:48,595 - test - INFO - listing zfs devices command zfs list -H -o name exit_status 0 output:
 zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15

2017-08-22 00:45:48,595 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:45:48,887 - test - INFO - destroy zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool destroy zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-22 00:45:48,888 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-22 00:45:49,243 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-22 00:45:49,244 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi
2017-08-22 00:45:49,814 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11; fi exit_status 0 output:
 
2017-08-22 00:45:49,814 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-22 00:45:50,300 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-22 00:45:50,301 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi
2017-08-22 00:45:50,906 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13; fi exit_status 0 output:
 
2017-08-22 00:45:50,907 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: systemctl disable zfs.target
2017-08-22 00:45:51,242 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command systemctl disable zfs.target exit_status 0 output:
 
2017-08-22 00:45:51,243 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi
2017-08-22 00:45:51,799 - test - INFO - create zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command if ! zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15; then zpool create -f zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 -o cachefile=none `ls -la /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk15 | awk '{print substr ($11, 7, 10)}'` && zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 && zpool import -d /dev/disk/by-id zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15; fi exit_status 0 output:
 
2017-08-22 00:45:51,800 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15
2017-08-22 00:45:52,126 - test - INFO - export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk15 exit_status 0 output:
 
2017-08-22 00:45:52,191 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-22 00:45:52,191 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:52,593 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: touch /tmp/chroma-agent-debug
2017-08-22 00:45:52,593 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:45:54,623 - test - DEBUG -     def supervisor_controlled_processes_running(self):
        # Use the api to verify the processes controlled by supervisor are all in a RUNNING state
        try:
            response = self.chroma_manager.get('/api/system_status/')
        except requests.ConnectionError:
            logger.warning("Connection error trying to connect to the manager")
            return False

        self.assertEqual(response.successful, True, response.text)
        system_status = response.json
        non_running_processes = []
        for process in system_status['supervisor']:
            if not process['statename'] == 'RUNNING':
                non_running_processes.append(process)

        if non_running_processes:
            logger.warning("Supervisor processes found not to be running: '%s'" % non_running_processes)
            return False
        else:
            return True
 evaluated to True
2017-08-22 00:45:55,651 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/1/'], u'complete': False, u'created_at': u'2017-08-22T07:45:54.714651', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-22 00:46:03,327 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/1/'], u'complete': True, u'created_at': u'2017-08-22T07:45:54.714651', u'message': u'Validation host configuration lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/1/', u'id': 1, u'logs': u''}
2017-08-22 00:46:03,399 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/2/'], u'complete': False, u'created_at': u'2017-08-22T07:45:54.804256', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-22 00:46:04,463 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/2/'], u'complete': True, u'created_at': u'2017-08-22T07:45:54.804256', u'message': u'Validation host configuration lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/2/', u'id': 2, u'logs': u''}
2017-08-22 00:46:06,351 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/3/'], u'complete': False, u'created_at': u'2017-08-22T07:46:04.410993', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-22 00:46:13,664 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/3/'], u'complete': True, u'created_at': u'2017-08-22T07:46:04.410993', u'message': u'Deploying agent to host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/3/', u'id': 3, u'logs': u''}
2017-08-22 00:46:13,703 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T07:46:05.132642', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-22 00:46:13,736 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/4/'], u'complete': True, u'created_at': u'2017-08-22T07:46:05.132642', u'message': u'Deploying agent to host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/4/', u'id': 4, u'logs': u''}
2017-08-22 00:46:14,277 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:15,340 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:16,389 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:17,442 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:18,497 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:19,543 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:20,585 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:21,635 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:22,697 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:23,749 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:24,799 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:25,848 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:26,889 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:27,942 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:28,992 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:30,043 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:31,090 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:32,144 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:33,193 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:34,244 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:35,295 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:36,348 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:37,402 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:38,450 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:39,499 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:40,553 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:41,602 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:42,652 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:43,697 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:44,752 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:45,802 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:46,846 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:47,900 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:48,953 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:50,010 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:51,065 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:52,113 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:53,161 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:54,210 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:55,260 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:56,308 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:57,362 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:46:58,416 - test - DEBUG -         self.wait_for_assert(lambda: self.assertTrue(self.get_host_validations(host).valid))
 tripped assertion: False is not true
2017-08-22 00:47:00,152 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': False, u'created_at': u'2017-08-22T07:46:58.875352', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u''}
2017-08-22 00:56:07,309 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/5/', u'/api/job/6/', u'/api/job/7/', u'/api/job/8/', u'/api/job/9/', u'/api/job/10/', u'/api/job/11/', u'/api/job/12/', u'/api/job/13/', u'/api/job/14/', u'/api/job/15/'], u'complete': True, u'created_at': u'2017-08-22T07:46:58.875352', u'message': u'Setup managed host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/5/', u'id': 5, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:56:07,399 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': False, u'created_at': u'2017-08-22T07:46:59.146156', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:56:10,739 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/16/', u'/api/job/17/', u'/api/job/18/', u'/api/job/19/', u'/api/job/20/', u'/api/job/21/', u'/api/job/22/', u'/api/job/23/', u'/api/job/24/', u'/api/job/25/', u'/api/job/26/'], u'complete': True, u'created_at': u'2017-08-22T07:46:59.146156', u'message': u'Setup managed host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/6/', u'id': 6, u'logs': u'\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'}
2017-08-22 00:56:11,410 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:56:11,410 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:56:11,853 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 00:56:11,854 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:56:12,938 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /etc/corosync/corosync.conf || true
2017-08-22 00:56:12,938 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:56:13,342 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat /etc/corosync/corosync.conf || true
2017-08-22 00:56:13,342 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:56:13,741 - test - DEBUG - set_mcast_port /api/corosync_configuration/1/ 52394
2017-08-22 00:56:14,127 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/28/', u'/api/job/29/', u'/api/job/30/', u'/api/job/31/', u'/api/job/32/'], u'complete': False, u'created_at': u'2017-08-22T07:56:13.175837', u'message': u'Update Corosync Configuration on host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u''}
2017-08-22 00:56:52,344 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/28/', u'/api/job/29/', u'/api/job/30/', u'/api/job/31/', u'/api/job/32/'], u'complete': True, u'created_at': u'2017-08-22T07:56:13.175837', u'message': u'Update Corosync Configuration on host lotus-59vm15.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/8/', u'id': 8, u'logs': u'\n\n\n\n'}
2017-08-22 00:56:52,345 - test - DEBUG - assertValue mcast_port /api/corosync_configuration/1/ 52394
2017-08-22 00:56:52,422 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /etc/corosync/corosync.conf || true
2017-08-22 00:56:52,422 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:56:53,000 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat /etc/corosync/corosync.conf || true
2017-08-22 00:56:53,001 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 00:56:53,348 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:56:54,401 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:56:55,440 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:56:56,480 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:56:57,521 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:56:58,568 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:56:59,608 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:00,649 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:01,690 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:02,729 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:03,768 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:04,808 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:05,848 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:06,884 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:07,933 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:08,976 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:10,014 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:11,054 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:12,090 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:13,126 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:14,164 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:15,205 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:16,247 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:17,288 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:18,329 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:19,370 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:20,411 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:21,451 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:22,491 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:23,530 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:24,571 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:25,612 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:26,653 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:27,691 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:28,731 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:29,771 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:30,810 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:31,851 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:32,889 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:33,930 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:34,968 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:36,007 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:37,045 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:38,085 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:39,124 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:40,162 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:41,201 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:42,238 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:43,272 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:44,316 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:45,357 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:46,399 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:47,436 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:48,474 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:49,512 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:50,554 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:51,595 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:52,646 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:53,684 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:54,723 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:55,762 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:56,800 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:57,840 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:58,878 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:57:59,918 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:00,956 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:01,995 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:03,034 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:04,075 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:05,115 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:06,152 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:07,193 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:08,232 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:09,274 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:10,316 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:11,354 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:12,392 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:13,425 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:14,462 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:15,495 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:16,533 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:17,570 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:18,611 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:19,643 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:20,686 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:21,724 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:22,759 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:23,796 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:24,832 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:25,869 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:26,905 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:27,944 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:28,981 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:30,016 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:31,048 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:32,086 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:33,124 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:34,162 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:35,200 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:36,241 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:37,282 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:38,319 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:39,354 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:40,396 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:41,436 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:42,476 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:43,509 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:44,548 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:45,586 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:46,626 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:47,665 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:48,700 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:49,737 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:50,779 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:51,818 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:52,854 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:53,893 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:54,928 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:55,966 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:57,004 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:58,046 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:58:59,087 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:00,127 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:01,168 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:02,207 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:03,248 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:04,289 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:05,330 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:06,370 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:07,410 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:08,450 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:09,490 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:10,530 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:11,570 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:12,611 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:13,652 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/1/' unexpectedly found in [u'/api/host/1/']
2017-08-22 00:59:14,757 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:15,796 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:16,835 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:17,870 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:18,911 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:19,950 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:20,987 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:22,027 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:23,068 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:24,107 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:25,146 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:26,184 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:27,222 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:28,260 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:29,287 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:30,312 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:31,351 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:32,388 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:33,427 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:34,475 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:35,517 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:36,559 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:37,598 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:38,638 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:39,675 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:40,716 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:41,754 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:42,793 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:43,835 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:44,876 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:45,918 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:46,958 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:48,003 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:49,042 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:50,085 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:51,127 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:52,167 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:53,208 - test - DEBUG -                 self.wait_for_assert(lambda: self.assertNoAlerts(server['resource_uri'], of_type='HostOfflineAlert'))
 tripped assertion: u'/api/host/2/' unexpectedly found in [u'/api/host/2/']
2017-08-22 00:59:54,249 - test - DEBUG - set_mcast_port /api/corosync_configuration/2/ 52394
2017-08-22 00:59:54,553 - test - DEBUG - wait_for_command: {u'jobs': [u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/'], u'complete': False, u'created_at': u'2017-08-22T07:59:53.620890', u'message': u'Update Corosync Configuration on host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u''}
2017-08-22 01:00:30,609 - test - DEBUG - command for None complete: {u'jobs': [u'/api/job/33/', u'/api/job/34/', u'/api/job/35/', u'/api/job/36/', u'/api/job/37/'], u'complete': True, u'created_at': u'2017-08-22T07:59:53.620890', u'message': u'Update Corosync Configuration on host lotus-59vm16.lotus.hpdd.lab.intel.com', u'cancelled': False, u'errored': False, u'resource_uri': u'/api/command/9/', u'id': 9, u'logs': u'\n\n\n\n'}
2017-08-22 01:00:30,610 - test - DEBUG - assertValue mcast_port /api/corosync_configuration/2/ 52394
2017-08-22 01:00:30,682 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat /etc/corosync/corosync.conf || true
2017-08-22 01:00:30,683 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:31,134 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat /etc/corosync/corosync.conf || true
2017-08-22 01:00:31,135 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:31,591 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:00:31,591 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:32,042 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_mcast_changes (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 01:00:32,042 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:32,445 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:00:32,445 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:32,822 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== stopping test test_mcast_changes (tests.integration.shared_storage_configuration.test_corosync.TestCorosync) ====="
2017-08-22 01:00:32,823 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:33,428 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:00:33,429 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:00:33,429 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:33,832 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-22 01:00:33,832 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
2017-08-22 01:00:33,832 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:34,227 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:00:34,227 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:00:34,227 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:34,600 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-22 01:00:34,600 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
2017-08-22 01:00:34,601 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:34,984 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:00:34,985 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:00:34,985 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:35,357 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-22 01:00:35,358 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
2017-08-22 01:00:35,358 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:35,732 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:00:35,733 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:00:35,733 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:36,102 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-22 01:00:36,102 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
2017-08-22 01:00:36,102 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:36,380 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:00:36,380 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:00:36,381 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:36,782 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-22 01:00:36,783 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
2017-08-22 01:00:36,783 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:37,159 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:00:37,160 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:00:37,160 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:37,547 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-22 01:00:37,548 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
2017-08-22 01:00:37,548 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:37,939 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:00:37,939 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:38,707 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 01:00:38,707 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:39,396 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-22 01:00:39,396 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:40,091 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:00:40,092 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:40,466 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 01:00:40,467 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:40,854 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 01:00:40,854 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:41,229 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:00:41,230 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:41,605 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 01:00:41,606 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:41,981 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 01:00:41,981 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:00:42,368 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-22 01:01:09,760 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-22 01:01:11,504 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-22 01:01:13,286 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-22 01:02:30,504 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-22 01:02:32,426 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-22 01:02:33,005 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-22 01:02:33,005 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-22 01:02:33,005 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-22 01:02:37,062 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:02:37,062 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:37,461 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:02:37,461 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:39,019 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:02:39,019 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:39,416 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:02:39,417 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:40,971 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:02:40,971 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:41,371 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:02:41,371 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:41,755 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:02:41,755 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:42,159 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:02:42,159 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:42,548 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:02:42,548 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:42,918 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:02:42,918 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:43,304 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:02:43,304 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:43,677 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:02:43,677 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:44,049 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 01:02:44,049 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:44,506 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 01:02:44,507 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:47,419 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 01:02:47,419 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:48,721 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 01:02:48,721 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:50,080 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:02:50,081 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:50,483 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 01:02:50,484 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:50,875 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 01:02:50,875 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:51,329 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 01:02:51,329 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:51,713 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 01:02:51,714 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:52,116 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 01:02:52,116 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:53,711 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 01:02:53,711 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:55,002 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 01:02:55,002 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:56,278 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:02:56,279 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:56,654 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 01:02:56,654 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:57,036 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 01:02:57,037 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:57,464 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 01:02:57,465 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:57,867 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 01:02:57,868 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:02:58,271 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 01:02:58,272 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:08:09,552 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 01:08:09,553 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:08:10,842 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 01:08:10,843 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:08:11,985 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:08:11,985 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:08:12,388 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 01:08:12,388 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:08:12,770 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 01:08:12,770 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:08:13,206 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 01:08:13,207 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:08:13,582 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 01:08:13,582 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:08:13,981 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 01:08:13,982 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:14,196 - test - ERROR - Traceback (most recent call last):
  File "/usr/share/chroma-manager/tests/integration/core/api_testcase_with_test_reset.py", line 137, in setUp
    self.reset_cluster()
  File "/usr/share/chroma-manager/tests/integration/core/api_testcase_with_test_reset.py", line 620, in reset_cluster
    self.remote_operations.clear_ha(self.TEST_SERVERS)
  File "/usr/share/chroma-manager/tests/integration/core/remote_operations.py", line 1200, in clear_ha
    clear_ha_script.read()))
  File "/usr/share/chroma-manager/tests/integration/core/remote_operations.py", line 470, in _ssh_address
    stdout = channel.makefile('rb').read()
  File "/usr/lib/python2.7/site-packages/paramiko/file.py", line 150, in read
    new_data = self._read(self._DEFAULT_BUFSIZE)
  File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 1234, in _read
    return self.channel.recv(size)
  File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv
    raise socket.timeout()
timeout: 
-------------------- >> begin captured logging << --------------------
test: INFO: Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: lotus-59vm15.lotus.hpdd.lab.intel.com is running
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: lotus-59vm16.lotus.hpdd.lab.intel.com is running
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: lotus-59vm17.lotus.hpdd.lab.intel.com is running
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
test: DEBUG: remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
test: INFO: SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: lotus-59vm18.lotus.hpdd.lab.intel.com is running
test: DEBUG: remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
test: INFO: SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: lotus-59vm19.lotus.hpdd.lab.intel.com is running
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: INFO: lotus-59vm19.lotus.hpdd.lab.intel.com is running
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_login (tests.integration.shared_storage_configuration.test_example_api_client.TestExampleApiClient) ====="
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
test: INFO: SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
test: INFO: SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
test: INFO: SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
test: DEBUG: remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
test: DEBUG: remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
test: DEBUG: remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
test: DEBUG: remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
test: DEBUG: remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
test: DEBUG: Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

test: DEBUG: Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
test: DEBUG: remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
test: INFO: SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
test: INFO: SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
test: INFO: SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
test: INFO: SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
test: INFO: SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
test: INFO: SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
test: INFO: SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
test: DEBUG: remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

test: INFO: SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
--------------------- >> end captured logging << ---------------------

2017-08-22 01:18:14,264 - test - INFO - Checking that lotus-59vm15.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:18:14,264 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:18:14,265 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:14,727 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com is running
2017-08-22 01:18:14,727 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: logger "==== starting test test_same_name (tests.integration.shared_storage_configuration.test_filesystem_same_name.TestFilesystemSameNameHYD832) ====="
2017-08-22 01:18:14,728 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:15,131 - test - INFO - Checking that lotus-59vm16.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:18:15,131 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:18:15,131 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:15,422 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com is running
2017-08-22 01:18:15,423 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: logger "==== starting test test_same_name (tests.integration.shared_storage_configuration.test_filesystem_same_name.TestFilesystemSameNameHYD832) ====="
2017-08-22 01:18:15,423 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:15,810 - test - INFO - Checking that lotus-59vm17.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:18:15,811 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:18:15,811 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:16,210 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com is running
2017-08-22 01:18:16,210 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: logger "==== starting test test_same_name (tests.integration.shared_storage_configuration.test_filesystem_same_name.TestFilesystemSameNameHYD832) ====="
2017-08-22 01:18:16,211 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:16,602 - test - INFO - Checking that lotus-59vm18.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:18:16,602 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:18:16,603 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:17,017 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com is running
2017-08-22 01:18:17,018 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: logger "==== starting test test_same_name (tests.integration.shared_storage_configuration.test_filesystem_same_name.TestFilesystemSameNameHYD832) ====="
2017-08-22 01:18:17,018 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:17,417 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:18:17,417 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:18:17,418 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:17,823 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-22 01:18:17,824 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_same_name (tests.integration.shared_storage_configuration.test_filesystem_same_name.TestFilesystemSameNameHYD832) ====="
2017-08-22 01:18:17,824 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:18,211 - test - INFO - Checking that lotus-59vm19.lotus.hpdd.lab.intel.com is running and restarting if necessary...
2017-08-22 01:18:18,211 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:18:18,212 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:18,598 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is running
2017-08-22 01:18:18,598 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: logger "==== starting test test_same_name (tests.integration.shared_storage_configuration.test_filesystem_same_name.TestFilesystemSameNameHYD832) ====="
2017-08-22 01:18:18,599 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:18,971 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:18:18,971 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:19,678 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 01:18:19,679 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:20,373 - test - DEBUG - remote_command[lotus-59vm12.lotus.hpdd.lab.intel.com]: mount
2017-08-22 01:18:20,374 - test - INFO - SSH address = lotus-59vm12.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:21,073 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:18:21,073 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:21,475 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 01:18:21,475 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:21,873 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 01:18:21,873 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:22,247 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:18:22,247 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:22,620 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sed -i '/lustre/d' /etc/fstab
2017-08-22 01:18:22,620 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:22,995 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: mount
2017-08-22 01:18:22,996 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:18:23,374 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config stop
2017-08-22 01:18:48,658 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service chroma-supervisor status
2017-08-22 01:18:50,237 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: service postgresql stop && rm -fr /var/lib/pgsql/data/*
2017-08-22 01:18:51,928 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: chroma-config setup admin lustre localhost --no-dbspace-check &> config_setup.log
2017-08-22 01:20:09,699 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for bundle_meta in /var/lib/chroma/repo/*/7/meta; do chroma-config bundle register $(dirname $bundle_meta); done &> config_bundle.log
2017-08-22 01:20:11,512 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: ls /tmp/iml-*/
2017-08-22 01:20:12,123 - test - DEBUG - Installer contents: base_managed_RH7.profile
base_monitored.profile
blockdevice_zfs.log
create_installer
default.profile
el7
EULA.txt
install
lesskey.out
posix_copytool_worker.profile
robinhood_server.profile

2017-08-22 01:20:12,123 - test - DEBUG - Found these profiles: base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile
2017-08-22 01:20:12,123 - test - DEBUG - remote_command[lotus-59vm13.lotus.hpdd.lab.intel.com]: for profile_pat in base_managed_RH7.profile base_monitored.profile default.profile posix_copytool_worker.profile robinhood_server.profile; do chroma-config profile register /tmp/iml-*/$profile_pat; done &> config_profile.log
2017-08-22 01:20:16,085 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:20:16,085 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:16,531 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:20:16,531 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:16,914 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:20:16,915 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:17,310 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:20:17,310 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:17,698 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:20:17,698 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:18,085 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:20:18,085 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:18,461 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:20:18,461 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:18,816 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:20:18,816 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:19,167 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:20:19,168 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:19,558 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:20:19,558 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:19,943 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: which chroma-agent
2017-08-22 01:20:19,943 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:20,311 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                    systemctl stop chroma-agent
                    i=0
                    
                    while systemctl status chroma-agent && [ "$i" -lt 600 ]; do
                        ((i++))
                        sleep 1
                    done
                    
                    if [ "$i" -eq 600 ]; then 
                        exit 1
                    fi
                    
2017-08-22 01:20:20,311 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:20,704 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 01:20:20,705 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:21,102 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 01:20:21,102 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:21,579 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 01:20:21,580 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:22,889 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 01:20:22,890 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:24,206 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:20:24,206 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:24,572 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 01:20:24,572 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:24,998 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 01:20:24,998 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:25,450 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 01:20:25,450 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:25,827 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 01:20:25,828 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:26,231 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 01:20:26,232 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:26,723 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 01:20:26,723 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:28,014 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 01:20:28,014 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:29,282 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:20:29,282 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:29,662 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 01:20:29,663 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:30,052 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 01:20:30,052 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:30,496 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 01:20:30,496 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:30,883 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 01:20:30,883 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:31,287 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 01:20:31,287 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:31,762 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 01:20:31,762 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:33,005 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 01:20:33,005 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:34,297 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:20:34,297 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:34,689 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 01:20:34,689 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:35,064 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 01:20:35,064 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:35,489 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 01:20:35,490 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:35,860 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: which crmadmin
2017-08-22 01:20:35,860 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:20:36,250 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: ring1_iface=eth1
#!/bin/sh

set -e

systemctl stop corosync pacemaker pcsd
systemctl disable corosync pacemaker pcsd

# figure it out for ourselves if we can
# otherwise the caller needs to have set it
if [ -f /etc/corosync/corosync.conf ]; then
    ring1_iface=$(ip route get $(sed -ne '/ringnumber: 1/{s///;n;s/.*: //p}' /etc/corosync/corosync.conf) | sed -ne 's/.* dev \(.*\)  *src.*/\1/p')
fi

ifconfig $ring1_iface 0.0.0.0 down

rm -f /etc/sysconfig/network-scripts/ifcfg-$ring1_iface
rm -f /etc/corosync/corosync.conf
rm -f /var/lib/pacemaker/cib/*
rm -f /var/lib/corosync/*

exit 0

2017-08-22 01:20:36,251 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:15,581 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=22/tcp $i; done
2017-08-22 01:28:15,582 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:16,826 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: for i in "" "--permanent"; do /usr/bin/firewall-cmd --add-port=988/tcp $i; done
2017-08-22 01:28:16,826 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:18,092 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: umount -t lustre -a
2017-08-22 01:28:18,093 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:18,524 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rpm -q chroma-agent
2017-08-22 01:28:18,525 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:18,974 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: service chroma-agent stop
2017-08-22 01:28:18,974 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:19,410 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                        rm -rf /var/lib/chroma/*;
                        
2017-08-22 01:28:19,411 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:19,773 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-22 01:28:19,773 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com is configured as a worker -- skipping.
2017-08-22 01:28:19,773 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 01:28:19,774 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:20,229 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 01:28:20,230 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:20,632 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 01:28:20,632 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:21,038 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 01:28:21,038 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:21,442 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 01:28:21,442 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:21,821 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: 
                              [ -f /etc/modprobe.d/iml_lnet_module_parameters.conf ] &&
                              rm -f /etc/modprobe.d/iml_lnet_module_parameters.conf &&
                              reboot &&
                              sleep 20
                              
2017-08-22 01:28:21,821 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:22,205 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:28:22,205 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:22,607 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:28:22,607 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:23,004 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:28:23,004 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:23,402 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:28:23,403 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:23,806 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:28:23,806 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:24,207 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: echo 'Checking if node is ready to receive commands.'
2017-08-22 01:28:24,207 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:24,582 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 01:28:24,582 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:24,984 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 01:28:24,984 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:25,383 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 01:28:25,383 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:25,786 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 01:28:25,786 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:26,184 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 01:28:26,184 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:26,573 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: rm -f /etc/chroma.cfg
2017-08-22 01:28:26,573 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:26,956 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 01:28:26,957 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:27,336 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 01:28:27,336 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:27,738 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 01:28:27,738 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:28,137 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: cat > /etc/chroma.cfg
2017-08-22 01:28:28,137 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:28,540 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 01:28:28,540 - test - INFO - SSH address = lotus-59vm15.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:28,942 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 01:28:28,942 - test - INFO - SSH address = lotus-59vm16.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:29,340 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 01:28:29,340 - test - INFO - SSH address = lotus-59vm17.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:29,714 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 01:28:29,714 - test - INFO - SSH address = lotus-59vm18.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:30,117 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 01:28:30,117 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:30,519 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: sync; sync
2017-08-22 01:28:30,519 - test - INFO - SSH address = lotus-59vm19.lotus.hpdd.lab.intel.com, args = {'username': 'root'}
2017-08-22 01:28:30,894 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 01:28:30,896 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 01:28:30,898 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 01:28:30,899 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 01:28:30,903 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 01:28:30,904 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync
2017-08-22 01:28:31,326 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:31,327 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:31,411 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:31,434 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:31,438 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:31,496 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk12 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:31,497 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 01:28:31,498 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 01:28:31,499 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 01:28:31,500 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 01:28:31,502 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 01:28:31,504 - test - DEBUG - remote_command[lotus-59vm19.lotus.hpdd.lab.intel.com]: dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync
2017-08-22 01:28:31,858 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:31,885 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:31,925 - test - INFO - lotus-59vm19.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:31,944 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:32,000 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:32,002 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: clear block device zpool(/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14) command dd if=/dev/zero of=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk14 bs=4k count=1; sync exit_status 0 output:
 
2017-08-22 01:28:32,003 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 01:28:32,004 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 01:28:32,005 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 01:28:32,007 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: modprobe zfs
2017-08-22 01:28:32,394 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 01:28:32,494 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 01:28:32,497 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 01:28:32,562 - test - INFO - lotus-59vm18.lotus.hpdd.lab.intel.com: checking for zfs presence command modprobe zfs exit_status 0 output:
 
2017-08-22 01:28:32,563 - test - DEBUG - remote_command[lotus-59vm15.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 01:28:32,564 - test - DEBUG - remote_command[lotus-59vm16.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 01:28:32,565 - test - DEBUG - remote_command[lotus-59vm17.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 01:28:32,566 - test - DEBUG - remote_command[lotus-59vm18.lotus.hpdd.lab.intel.com]: zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11
2017-08-22 01:28:32,921 - test - INFO - lotus-59vm15.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 0 output:
 
2017-08-22 01:28:32,994 - test - INFO - lotus-59vm16.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output:
 
2017-08-22 01:28:32,996 - test - INFO - lotus-59vm17.lotus.hpdd.lab.intel.com: export zfs device zpool(zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11) command zpool export zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk11 exit_status 1 output: