Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1509471 - Installation failed: socket is already closed: port 9000
Summary: Installation failed: socket is already closed: port 9000
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-zaqar
Version: 12.0 (Pike)
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ga
: ---
Assignee: Thomas Hervé
QA Contact: Shai Revivo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-03 21:04 UTC by Steve Reichard
Modified: 2017-11-09 21:24 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-09 21:24:15 UTC


Attachments (Terms of Use)

Description Steve Reichard 2017-11-03 21:04:30 UTC
Description of problem:

Attempts to install OSP12 have not worked for me since the last week of October:

(undercloud) [stack@ospha-inst ~]$ ./deploy-overcloud.sh |& tee deployover3.out
+ PROJECT=ospha
+ date
Fri Nov  3 19:26:07 UTC 2017
+ source stackrc
+++ set
+++ awk '{FS="="}  /^OS_/ {print $1}'
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_AUTH_TYPE
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_AUTH_URL
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_BAREMETAL_API_VERSION
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_CLOUDNAME
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_IDENTITY_API_VERSION
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_NO_CACHE
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_PASSWORD
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_PROJECT_DOMAIN_NAME
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_PROJECT_NAME
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_USERNAME
++ for key in '$( set | awk '\''{FS="="}  /^OS_/ {print $1}'\'' )'
++ unset OS_USER_DOMAIN_NAME
++ NOVA_VERSION=1.1
++ export NOVA_VERSION
+++ sudo hiera admin_password
++ OS_PASSWORD=changeme
++ export OS_PASSWORD
++ OS_AUTH_TYPE=password
++ export OS_AUTH_TYPE
++ OS_AUTH_URL=https://10.19.139.27:13000/
++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'
++ export OS_AUTH_URL
++ export PYTHONWARNINGS
++ OS_USERNAME=admin
++ OS_PROJECT_NAME=admin
++ COMPUTE_API_VERSION=1.1
++ IRONIC_API_VERSION=1.34
++ OS_BAREMETAL_API_VERSION=1.34
++ OS_NO_CACHE=True
++ OS_CLOUDNAME=undercloud
++ export OS_USERNAME
++ export OS_PROJECT_NAME
++ export COMPUTE_API_VERSION
++ export IRONIC_API_VERSION
++ export OS_BAREMETAL_API_VERSION
++ export OS_NO_CACHE
++ export OS_CLOUDNAME
++ OS_IDENTITY_API_VERSION=3
++ export OS_IDENTITY_API_VERSION
++ OS_PROJECT_DOMAIN_NAME=Default
++ export OS_PROJECT_DOMAIN_NAME
++ OS_USER_DOMAIN_NAME=Default
++ export OS_USER_DOMAIN_NAME
++ '[' -z 1 ']'
+ openstack overcloud deploy --templates -e /home/stack/mytemplates/env/01-tls.yaml -e /home/stack/mytemplates/env/90-ospha-rhos12-images.yaml -e /home/stack/mytemplates/env/99-ospha-misc.yaml --debug --roles-file /home/stack/mytemplates/ospha-roles-data.yaml --stack ospha --log-file overcloudDeploy.log
START with options: [u'overcloud', u'deploy', u'--templates', u'-e', u'/home/stack/mytemplates/env/01-tls.yaml', u'-e', u'/home/stack/mytemplates/env/90-ospha-rhos12-images.yaml', u'-e', u'/home/stack/mytemplates/env/99-ospha-misc.yaml', u'--debug', u'--roles-file', u'/home/stack/mytemplates/ospha-roles-data.yaml', u'--stack', u'ospha', u'--log-file', u'overcloudDeploy.log']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', auth_type='password', auth_url='https://10.19.139.27:13000/', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=u'overcloudDeploy.log', openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.34', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_event_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', roles='', service_provider_endpoint='', service_provider_entity_id='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://10.19.139.27:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, 'log_file': u'overcloudDeploy.log', u'interface': None, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://10.19.139.27:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, 'log_file': u'overcloudDeploy.log', u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
orchestration API version 1, cmd group openstack.orchestration.v1
messaging API version 2, cmd group openstack.messaging.v2
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
baremetal API version 1.34, cmd group openstack.baremetal.v1
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
event API version 2, cmd group openstack.event.v2
key_manager API version 1, cmd group openstack.key_manager.v1
dns API version 2, cmd group openstack.dns.v2
container_infra API version 1, cmd group openstack.container_infra.v1
database API version 1, cmd group openstack.database.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://10.19.139.27:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, 'log_file': u'overcloudDeploy.log', u'interface': None, u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://10.19.139.27:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, 'log_file': u'overcloudDeploy.log', u'interface': None, u'disable_vendor_agent': {}}
command: overcloud deploy -> tripleoclient.v1.overcloud_deploy.DeployOvercloud (auth=True)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://10.19.139.27:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, 'log_file': u'overcloudDeploy.log', u'interface': None, u'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'https://10.19.139.27:13000/', 'password': '***', 'project_domain_name': 'Default'}
Get auth_ref
REQ: curl -g -i -X GET https://10.19.139.27:13000/ -H "Accept: application/json" -H "User-Agent: osc-lib/1.7.0 keystoneauth1/3.1.0 python-requests/2.11.1 CPython/2.7.5"
Starting new HTTPS connection (1): 10.19.139.27
"GET / HTTP/1.1" 300 602
RESP: [300] Date: Fri, 03 Nov 2017 19:26:09 GMT Server: Apache Vary: X-Auth-Token Content-Length: 602 Content-Type: application/json 
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "https://10.19.139.27:13000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://10.19.139.27:13000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

Making authentication request to https://10.19.139.27:13000/v3/auth/tokens
"POST /v3/auth/tokens HTTP/1.1" 201 10374
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "e17bda95c1de466c804e4d101b37f363", "name": "admin"}], "expires_at": "2017-11-03T23:26:10.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "ac59775b9fd343f395a3cc0997a3cbde", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.9.3:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "3ae52c1fe70e49fda108977c3abb513f"}, {"url": "http://192.168.9.3:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a65e415d7ff6430fa7b0ad64536353b2"}, {"url": "https://10.19.139.27:13696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "fdfd7b686e1c40aaa29c4ecf67fa6bdd"}], "type": "network", "id": "0f9254e4c0cf48d1bb4f0edd7b4b395f", "name": "neutron"}, {"endpoints": [{"url": "https://10.19.139.27:13774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "366334eec54e4ffbbb6e47fdc094ac0a"}, {"url": "http://192.168.9.3:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "b1052b261d6e4015b9b0a18346613b2c"}, {"url": "http://192.168.9.3:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "f2b04255bbd5407ea24e3321cf3ac67d"}], "type": "compute", "id": "162418ee2f7441ad8d0d220fc7d17a79", "name": "nova"}, {"endpoints": [{"url": "http://192.168.9.3:8777", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "9c34c47b3a8843c8a59d2b8888ff2448"}, {"url": "https://10.19.139.27:13777", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a561172a3f154819961dfefc24bdada7"}, {"url": "http://192.168.9.3:8777", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "c6ff6e154ce94b47bb3402e8b7f75974"}], "type": "metering", "id": "282a5a26fd6a47b99bb7a3848398088f", "name": "ceilometer"}, {"endpoints": [{"url": "ws://192.168.9.3:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "262b8920f05048cda73616839ba28880"}, {"url": "wss://10.19.139.27:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a2818c2c44a941a99867bf86d68a56a4"}, {"url": "ws://192.168.9.3:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ed8093e2e8ef4fb5bde891e352c7d741"}], "type": "messaging-websocket", "id": "3bbd8cf8a82440dba664bb165f4e94b9", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://192.168.9.3:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "265bdf75eab546a5a5b1978f9dd14818"}, {"url": "http://192.168.9.3:8080/v1/AUTH_ac59775b9fd343f395a3cc0997a3cbde", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "338daff9256a4c9ba84dbdcab8b5ac39"}, {"url": "https://10.19.139.27:13808/v1/AUTH_ac59775b9fd343f395a3cc0997a3cbde", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "442662c326184f8ab58f92f740ae61c3"}], "type": "object-store", "id": "3be985d2c2ed4b8f8a7d598b332a87a8", "name": "swift"}, {"endpoints": [{"url": "http://192.168.9.3:8000/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "4bdc407839774a578654892fc6b7ef18"}, {"url": "http://192.168.9.3:8000/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "943a7ec0a1c842549fa28afa1b079e1c"}, {"url": "https://10.19.139.27:13800/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "966269a4b1ae4eecab0ecd0aad8d0709"}], "type": "cloudformation", "id": "4584fe3cc0124c7e8da43311d3b79170", "name": "heat-cfn"}, {"endpoints": [{"url": "https://10.19.139.27:13042", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "595bed114ab34d078033fcafc53c9a99"}, {"url": "http://192.168.9.3:8042", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6533d65e70eb4cb0b0eeb814a938612b"}, {"url": "http://192.168.9.3:8042", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ae92bedfedc545a8b1761be9b07f167e"}], "type": "alarming", "id": "50ae6db289114a458f13b1d82429435d", "name": "aodh"}, {"endpoints": [{"url": "http://192.168.9.3:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "791b64f068db42b1982ece164fc831a0"}, {"url": "https://10.19.139.27:13292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "92b2e330e1424314beba13d006d1be40"}, {"url": "http://192.168.9.3:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "e40d94ee969944af9486796a95bb1393"}], "type": "image", "id": "549cf3c9e6c14f06842ad058baac21db", "name": "glance"}, {"endpoints": [{"url": "https://10.19.139.27:13004/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "571257cfbbb44cf8878daeb74be9ec02"}, {"url": "http://192.168.9.3:8004/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "81c6bf9a7d934f57bf512182b820a128"}, {"url": "http://192.168.9.3:8004/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d64b108ad85c4e05ae39335df3be63c5"}], "type": "orchestration", "id": "56a289bf92b6431c998726bfdcdfa2e6", "name": "heat"}, {"endpoints": [{"url": "http://192.168.9.3:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a2ff354a0e4c447d8e281b7fe38153db"}, {"url": "https://10.19.139.27:13989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "b4f0848cb1db46929972225595647273"}, {"url": "http://192.168.9.3:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "d67d306bbab24bd787407765069f5c5f"}], "type": "workflowv2", "id": "71b283d71f254759b03f98b1369f23f6", "name": "mistral"}, {"endpoints": [{"url": "https://10.19.139.27:13050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2e656d6de56b43f8a7c7035a430246e1"}, {"url": "http://192.168.9.3:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "bfbaa21504504db3be257c46e0783b26"}, {"url": "http://192.168.9.3:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "d107086a003745ae8c7f833b10f8b864"}], "type": "baremetal-introspection", "id": "83902427c44a40d39da9b97f838bcc5c", "name": "ironic-inspector"}, {"endpoints": [{"url": "http://192.168.9.3:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0fdeabba5328471b8595074bac0e23d3"}, {"url": "https://10.19.139.27:13888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3cab597df87845af8b31a8ff8800ab46"}, {"url": "http://192.168.9.3:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a673e4400e96456ba7e88ce71d3e9aa9"}], "type": "messaging", "id": "acaa4ea7c0d540598e7e155bfd420d93", "name": "zaqar"}, {"endpoints": [{"url": "https://10.19.139.27:13000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2a136c699b7143158776c6c3ba58c8c7"}, {"url": "http://192.168.9.3:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "7956cf25708b410590e53d8553c9548b"}, {"url": "http://192.168.9.3:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "fc5618e7e2ce47b0a717555eeaa80f82"}], "type": "identity", "id": "ea1a6b615b4a4c17b5f40f11fe92798d", "name": "keystone"}, {"endpoints": [{"url": "https://10.19.139.27:13779", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "178ae3ab90a74325b0d40e62ef37f394"}, {"url": "http://192.168.9.3:8977", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "32aacfab56ce4967853c79b3d5347feb"}, {"url": "http://192.168.9.3:8977", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "d19270584ee6480ba3287b9554c4cf3a"}], "type": "event", "id": "ed2f046205004db9aab7c755db07d752", "name": "panko"}, {"endpoints": [{"url": "https://10.19.139.27:13041", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "09b873499a6f4a79b1b2461f21fe53df"}, {"url": "http://192.168.9.3:8041", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "1be463d058964b56aa353f2bb027e2ca"}, {"url": "http://192.168.9.3:8041", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "8e3e38cc5bc446b3b4c6c51bd1fef4a4"}], "type": "metric", "id": "f4bab78840eb43f1b86f5d30a4a0510c", "name": "gnocchi"}, {"endpoints": [{"url": "http://192.168.9.3:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "9b3a6e2670cf4927853462152eb6d7a8"}, {"url": "http://192.168.9.3:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a310027e77584757aeb93e29b23950e8"}, {"url": "https://10.19.139.27:13385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "f28bd8398e904eeeb4880138a8106057"}], "type": "baremetal", "id": "f71a1f6da0e543e08eb6ca73b9b27eb5", "name": "ironic"}, {"endpoints": [{"url": "http://192.168.9.3:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "0993dc54d3784e088d00f8407cf0a0a5"}, {"url": "https://10.19.139.27:13778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "1f974e4f38784ec78759a7d7a81f92aa"}, {"url": "http://192.168.9.3:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "919fa29a4aa14dcda2b528378121d006"}], "type": "placement", "id": "fada01f0dd964ed0964f2e446e977380", "name": "placement"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "0b5a174e1ad5421f90233732c05067f8"}, "audit_ids": ["65z65Tk0Sr6SZJqas47GpA"], "issued_at": "2017-11-03T19:26:10.000000Z"}}
run(Namespace(answers_file=None, block_storage_flavor=None, block_storage_scale=None, ceph_storage_flavor=None, ceph_storage_scale=None, compute_flavor=None, compute_scale=None, control_flavor=None, control_scale=None, deployed_server=False, disable_password_generation=False, disable_validations=False, dry_run=False, environment_directories=['/home/stack/.tripleo/environments'], environment_files=[u'/home/stack/mytemplates/env/01-tls.yaml', u'/home/stack/mytemplates/env/90-ospha-rhos12-images.yaml', u'/home/stack/mytemplates/env/99-ospha-misc.yaml'], force_postconfig=False, libvirt_type=None, networks_file=None, no_cleanup=False, no_proxy='', ntp_server=None, overcloud_ssh_user='heat-admin', plan_environment_file=None, reg_activation_key='', reg_force=False, reg_method='satellite', reg_org='', reg_sat_url='', rhel_reg=False, roles_file=u'/home/stack/mytemplates/ospha-roles-data.yaml', run_validations=False, skip_deploy_identifier=False, skip_postconfig=False, stack=u'ospha', swift_storage_flavor=None, swift_storage_scale=None, templates='/usr/share/openstack-tripleo-heat-templates/', timeout=240, update_plan_only=False, validation_errors_fatal=True, validation_warnings_fatal=False))
take_action(Namespace(answers_file=None, block_storage_flavor=None, block_storage_scale=None, ceph_storage_flavor=None, ceph_storage_scale=None, compute_flavor=None, compute_scale=None, control_flavor=None, control_scale=None, deployed_server=False, disable_password_generation=False, disable_validations=False, dry_run=False, environment_directories=['/home/stack/.tripleo/environments'], environment_files=[u'/home/stack/mytemplates/env/01-tls.yaml', u'/home/stack/mytemplates/env/90-ospha-rhos12-images.yaml', u'/home/stack/mytemplates/env/99-ospha-misc.yaml'], force_postconfig=False, libvirt_type=None, networks_file=None, no_cleanup=False, no_proxy='', ntp_server=None, overcloud_ssh_user='heat-admin', plan_environment_file=None, reg_activation_key='', reg_force=False, reg_method='satellite', reg_org='', reg_sat_url='', rhel_reg=False, roles_file=u'/home/stack/mytemplates/ospha-roles-data.yaml', run_validations=False, skip_deploy_identifier=False, skip_postconfig=False, stack=u'ospha', swift_storage_flavor=None, swift_storage_scale=None, templates='/usr/share/openstack-tripleo-heat-templates/', timeout=240, update_plan_only=False, validation_errors_fatal=True, validation_warnings_fatal=False))
Making authentication request to https://10.19.139.27:13000/v3/auth/tokens
"POST /v3/auth/tokens HTTP/1.1" 201 10374
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "e17bda95c1de466c804e4d101b37f363", "name": "admin"}], "expires_at": "2017-11-03T23:26:10.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "ac59775b9fd343f395a3cc0997a3cbde", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.9.3:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "3ae52c1fe70e49fda108977c3abb513f"}, {"url": "http://192.168.9.3:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a65e415d7ff6430fa7b0ad64536353b2"}, {"url": "https://10.19.139.27:13696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "fdfd7b686e1c40aaa29c4ecf67fa6bdd"}], "type": "network", "id": "0f9254e4c0cf48d1bb4f0edd7b4b395f", "name": "neutron"}, {"endpoints": [{"url": "https://10.19.139.27:13774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "366334eec54e4ffbbb6e47fdc094ac0a"}, {"url": "http://192.168.9.3:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "b1052b261d6e4015b9b0a18346613b2c"}, {"url": "http://192.168.9.3:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "f2b04255bbd5407ea24e3321cf3ac67d"}], "type": "compute", "id": "162418ee2f7441ad8d0d220fc7d17a79", "name": "nova"}, {"endpoints": [{"url": "http://192.168.9.3:8777", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "9c34c47b3a8843c8a59d2b8888ff2448"}, {"url": "https://10.19.139.27:13777", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a561172a3f154819961dfefc24bdada7"}, {"url": "http://192.168.9.3:8777", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "c6ff6e154ce94b47bb3402e8b7f75974"}], "type": "metering", "id": "282a5a26fd6a47b99bb7a3848398088f", "name": "ceilometer"}, {"endpoints": [{"url": "ws://192.168.9.3:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "262b8920f05048cda73616839ba28880"}, {"url": "wss://10.19.139.27:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a2818c2c44a941a99867bf86d68a56a4"}, {"url": "ws://192.168.9.3:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ed8093e2e8ef4fb5bde891e352c7d741"}], "type": "messaging-websocket", "id": "3bbd8cf8a82440dba664bb165f4e94b9", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://192.168.9.3:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "265bdf75eab546a5a5b1978f9dd14818"}, {"url": "http://192.168.9.3:8080/v1/AUTH_ac59775b9fd343f395a3cc0997a3cbde", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "338daff9256a4c9ba84dbdcab8b5ac39"}, {"url": "https://10.19.139.27:13808/v1/AUTH_ac59775b9fd343f395a3cc0997a3cbde", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "442662c326184f8ab58f92f740ae61c3"}], "type": "object-store", "id": "3be985d2c2ed4b8f8a7d598b332a87a8", "name": "swift"}, {"endpoints": [{"url": "http://192.168.9.3:8000/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "4bdc407839774a578654892fc6b7ef18"}, {"url": "http://192.168.9.3:8000/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "943a7ec0a1c842549fa28afa1b079e1c"}, {"url": "https://10.19.139.27:13800/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "966269a4b1ae4eecab0ecd0aad8d0709"}], "type": "cloudformation", "id": "4584fe3cc0124c7e8da43311d3b79170", "name": "heat-cfn"}, {"endpoints": [{"url": "https://10.19.139.27:13042", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "595bed114ab34d078033fcafc53c9a99"}, {"url": "http://192.168.9.3:8042", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6533d65e70eb4cb0b0eeb814a938612b"}, {"url": "http://192.168.9.3:8042", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ae92bedfedc545a8b1761be9b07f167e"}], "type": "alarming", "id": "50ae6db289114a458f13b1d82429435d", "name": "aodh"}, {"endpoints": [{"url": "http://192.168.9.3:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "791b64f068db42b1982ece164fc831a0"}, {"url": "https://10.19.139.27:13292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "92b2e330e1424314beba13d006d1be40"}, {"url": "http://192.168.9.3:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "e40d94ee969944af9486796a95bb1393"}], "type": "image", "id": "549cf3c9e6c14f06842ad058baac21db", "name": "glance"}, {"endpoints": [{"url": "https://10.19.139.27:13004/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "571257cfbbb44cf8878daeb74be9ec02"}, {"url": "http://192.168.9.3:8004/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "81c6bf9a7d934f57bf512182b820a128"}, {"url": "http://192.168.9.3:8004/v1/ac59775b9fd343f395a3cc0997a3cbde", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d64b108ad85c4e05ae39335df3be63c5"}], "type": "orchestration", "id": "56a289bf92b6431c998726bfdcdfa2e6", "name": "heat"}, {"endpoints": [{"url": "http://192.168.9.3:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a2ff354a0e4c447d8e281b7fe38153db"}, {"url": "https://10.19.139.27:13989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "b4f0848cb1db46929972225595647273"}, {"url": "http://192.168.9.3:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "d67d306bbab24bd787407765069f5c5f"}], "type": "workflowv2", "id": "71b283d71f254759b03f98b1369f23f6", "name": "mistral"}, {"endpoints": [{"url": "https://10.19.139.27:13050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2e656d6de56b43f8a7c7035a430246e1"}, {"url": "http://192.168.9.3:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "bfbaa21504504db3be257c46e0783b26"}, {"url": "http://192.168.9.3:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "d107086a003745ae8c7f833b10f8b864"}], "type": "baremetal-introspection", "id": "83902427c44a40d39da9b97f838bcc5c", "name": "ironic-inspector"}, {"endpoints": [{"url": "http://192.168.9.3:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0fdeabba5328471b8595074bac0e23d3"}, {"url": "https://10.19.139.27:13888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3cab597df87845af8b31a8ff8800ab46"}, {"url": "http://192.168.9.3:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a673e4400e96456ba7e88ce71d3e9aa9"}], "type": "messaging", "id": "acaa4ea7c0d540598e7e155bfd420d93", "name": "zaqar"}, {"endpoints": [{"url": "https://10.19.139.27:13000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2a136c699b7143158776c6c3ba58c8c7"}, {"url": "http://192.168.9.3:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "7956cf25708b410590e53d8553c9548b"}, {"url": "http://192.168.9.3:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "fc5618e7e2ce47b0a717555eeaa80f82"}], "type": "identity", "id": "ea1a6b615b4a4c17b5f40f11fe92798d", "name": "keystone"}, {"endpoints": [{"url": "https://10.19.139.27:13779", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "178ae3ab90a74325b0d40e62ef37f394"}, {"url": "http://192.168.9.3:8977", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "32aacfab56ce4967853c79b3d5347feb"}, {"url": "http://192.168.9.3:8977", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "d19270584ee6480ba3287b9554c4cf3a"}], "type": "event", "id": "ed2f046205004db9aab7c755db07d752", "name": "panko"}, {"endpoints": [{"url": "https://10.19.139.27:13041", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "09b873499a6f4a79b1b2461f21fe53df"}, {"url": "http://192.168.9.3:8041", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "1be463d058964b56aa353f2bb027e2ca"}, {"url": "http://192.168.9.3:8041", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "8e3e38cc5bc446b3b4c6c51bd1fef4a4"}], "type": "metric", "id": "f4bab78840eb43f1b86f5d30a4a0510c", "name": "gnocchi"}, {"endpoints": [{"url": "http://192.168.9.3:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "9b3a6e2670cf4927853462152eb6d7a8"}, {"url": "http://192.168.9.3:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a310027e77584757aeb93e29b23950e8"}, {"url": "https://10.19.139.27:13385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "f28bd8398e904eeeb4880138a8106057"}], "type": "baremetal", "id": "f71a1f6da0e543e08eb6ca73b9b27eb5", "name": "ironic"}, {"endpoints": [{"url": "http://192.168.9.3:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "0993dc54d3784e088d00f8407cf0a0a5"}, {"url": "https://10.19.139.27:13778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "1f974e4f38784ec78759a7d7a81f92aa"}, {"url": "http://192.168.9.3:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "919fa29a4aa14dcda2b528378121d006"}], "type": "placement", "id": "fada01f0dd964ed0964f2e446e977380", "name": "placement"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "0b5a174e1ad5421f90233732c05067f8"}, "audit_ids": ["JmGw3umXSquLNUvq39To9Q"], "issued_at": "2017-11-03T19:26:10.000000Z"}}
Instantiating workflow engine client: <class 'mistralclient.api.v2.client.Client'>
REQ: curl -g -i -X GET https://10.19.139.27:13000/ -H "User-Agent: osc-lib/1.7.0 keystoneauth1/3.1.0 python-requests/2.11.1 CPython/2.7.5" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}70363ac1fb074fb643d3dc7a4b4ca9fab1383e93"
"GET / HTTP/1.1" 300 602
RESP: [300] Date: Fri, 03 Nov 2017 19:26:10 GMT Server: Apache Vary: X-Auth-Token Content-Length: 602 Content-Type: application/json 
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "https://10.19.139.27:13000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://10.19.139.27:13000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

Instantiating orchestration client: <class 'heatclient.v1.client.Client'>
Instantiating compute client for API Version Major: 2, Minor: 0
Instantiating compute api: <class 'openstackclient.api.compute_v2.APIv2'>
Instantiating baremetal client: <class 'ironicclient.v1.client.Client'>
Baremetal API version: 1.9
Checking that the disable_upgrade_deployment flag is set at least once in the roles file
REQ: curl -g -i -X GET https://10.19.139.27:13004/v1/ac59775b9fd343f395a3cc0997a3cbde/stacks/ospha -H "User-Agent: python-heatclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}70363ac1fb074fb643d3dc7a4b4ca9fab1383e93"
Starting new HTTPS connection (1): 10.19.139.27
"GET /v1/ac59775b9fd343f395a3cc0997a3cbde/stacks/ospha HTTP/1.1" 404 590
RESP: [404] Date: Fri, 03 Nov 2017 19:26:11 GMT Server: Apache x-openstack-request-id: req-f2ba0023-7df5-478c-b5a7-556f97d082a5 Content-Length: 590 Content-Type: application/json 
RESP BODY: {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Stack (ospha) could not be found.", "traceback": "Traceback (most recent call last):\n\n  File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 409, in wrapped\n    return func(self, ctx, *args, **kwargs)\n\n  File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 487, in identify_stack\n    raise exception.EntityNotFound(entity='Stack', name=stack_name)\n\nEntityNotFound: The Stack (ospha) could not be found.\n", "type": "EntityNotFound"}, "title": "Not Found"}

GET call to orchestration for https://10.19.139.27:13004/v1/ac59775b9fd343f395a3cc0997a3cbde/stacks/ospha used request id req-f2ba0023-7df5-478c-b5a7-556f97d082a5
Starting _pre_verify_capabilities
REQ: curl -g -i -X POST https://10.19.139.27:13989/v2/executions -H "User-Agent: osc-lib/1.7.0 keystoneauth1/3.1.0 python-requests/2.11.1 CPython/2.7.5" -H "content-type: application/json" -H "X-Auth-Token: {SHA1}70363ac1fb074fb643d3dc7a4b4ca9fab1383e93" -d '{"input": "{\"run_validations\": true, \"roles_info\": {\"control\": [null, null], \"ceph-storage\": [null, null], \"block-storage\": [null, null], \"compute\": [null, null], \"swift-storage\": [null, null]}, \"parameters\": {\"StackAction\": \"CREATE\", \"UpdateIdentifier\": \"\"}, \"deploy_ramdisk_name\": \"bm-deploy-ramdisk\", \"queue_name\": \"e25b435f-3d39-4b24-9369-a4fec00653a0\", \"deploy_kernel_name\": \"bm-deploy-kernel\", \"default_role_counts\": {\"ControllerCount\": 1, \"Compute - ExternalCount\": \"1 - External\", \"CephStorageCount\": 0, \"BlockStorageCount\": 0, \"ObjectStorageCount\": 0}, \"stack_id\": \"ospha\"}", "workflow_name": "tripleo.validations.v1.check_pre_deployment_validations", "description": ""}'
Starting new HTTPS connection (1): 10.19.139.27
"POST /v2/executions HTTP/1.1" 201 1052
RESP: [201] Content-Length: 1052 Content-Type: application/json Date: Fri, 03 Nov 2017 19:26:15 GMT Connection: keep-alive 
RESP BODY: {"state_info": null, "created_at": "2017-11-03 19:26:15", "description": "", "state": "RUNNING", "workflow_name": "tripleo.validations.v1.check_pre_deployment_validations", "task_execution_id": null, "updated_at": "2017-11-03 19:26:15", "workflow_id": "694577fa-29b2-4f57-b282-d395ba334560", "params": "{\"namespace\": \"\"}", "workflow_namespace": "", "output": "{}", "input": "{\"run_validations\": true, \"roles_info\": {\"control\": [null, null], \"compute\": [null, null], \"block-storage\": [null, null], \"ceph-storage\": [null, null], \"swift-storage\": [null, null]}, \"parameters\": {\"StackAction\": \"CREATE\", \"UpdateIdentifier\": \"\"}, \"deploy_ramdisk_name\": \"bm-deploy-ramdisk\", \"queue_name\": \"e25b435f-3d39-4b24-9369-a4fec00653a0\", \"deploy_kernel_name\": \"bm-deploy-kernel\", \"default_role_counts\": {\"ControllerCount\": 1, \"Compute - ExternalCount\": \"1 - External\", \"CephStorageCount\": 0, \"BlockStorageCount\": 0, \"ObjectStorageCount\": 0}, \"stack_id\": \"ospha\"}", "id": "04f4e6d5-602f-4729-9df7-19f598485782"}

HTTP POST https://10.19.139.27:13989/v2/executions 201
Instantiating messaging websocket client: wss://10.19.139.27:9000
Waiting for messages on queue 'e25b435f-3d39-4b24-9369-a4fec00653a0' with no timeout.
socket is already closed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 137, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 864, in take_action
    stack, parameters, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 639, in _predeploy_verify_capabilities
    **validation_params
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/validations.py", line 35, in check_predeployment_validations
    warnings += payload['warnings']
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 160, in __exit__
    self.cleanup()
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 104, in cleanup
    self.send('queue_delete', {'queue_name': self._queue_name})
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 120, in send
    self._ws.send(json.dumps(msg))
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 289, in send
    return self.send_frame(frame)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 314, in send_frame
    l = self._send(data)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 472, in _send
    return send(self.sock, data)
  File "/usr/lib/python2.7/site-packages/websocket/_socket.py", line 109, in send
    raise WebSocketConnectionClosedException("socket is already closed.")
WebSocketConnectionClosedException: socket is already closed.
clean_up DeployOvercloud: socket is already closed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 137, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 864, in take_action
    stack, parameters, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 639, in _predeploy_verify_capabilities
    **validation_params
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/validations.py", line 35, in check_predeployment_validations
    warnings += payload['warnings']
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 160, in __exit__
    self.cleanup()
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 104, in cleanup
    self.send('queue_delete', {'queue_name': self._queue_name})
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 120, in send
    self._ws.send(json.dumps(msg))
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 289, in send
    return self.send_frame(frame)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 314, in send_frame
    l = self._send(data)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 472, in _send
    return send(self.sock, data)
  File "/usr/lib/python2.7/site-packages/websocket/_socket.py", line 109, in send
    raise WebSocketConnectionClosedException("socket is already closed.")
WebSocketConnectionClosedException: socket is already closed.

END return value: 1
Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: 04f4e6d5-602f-4729-9df7-19f598485782
+ date
Fri Nov  3 20:26:31 UTC 2017
(undercloud) [stack@ospha-inst ~]$ 


(undercloud) [stack@ospha-inst ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:ad:36:9a brd ff:ff:ff:ff:ff:ff
    inet 10.19.139.30/21 brd 10.19.143.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2620:52:0:1388:5054:ff:fead:369a/64 scope global mngtmpaddr dynamic 
       valid_lft 2591789sec preferred_lft 604589sec
    inet6 fe80::5054:ff:fead:369a/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000
    link/ether 52:54:00:81:db:29 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe81:db29/64 scope link 
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether aa:9f:c3:89:ab:bb brd ff:ff:ff:ff:ff:ff
5: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether 52:54:00:81:db:29 brd ff:ff:ff:ff:ff:ff
    inet 192.168.9.1/24 brd 192.168.9.255 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 192.168.9.3/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet 10.19.139.27/32 scope global br-ctlplane
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe81:db29/64 scope link 
       valid_lft forever preferred_lft forever
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:74:d9:51:bb brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 3e:8b:ae:07:3e:4a brd ff:ff:ff:ff:ff:ff
(undercloud) [stack@ospha-inst ~]$ ping 10.19.139.27
PING 10.19.139.27 (10.19.139.27) 56(84) bytes of data.
64 bytes from 10.19.139.27: icmp_seq=1 ttl=64 time=0.072 ms
64 bytes from 10.19.139.27: icmp_seq=2 ttl=64 time=0.063 ms
64 bytes from 10.19.139.27: icmp_seq=3 ttl=64 time=0.062 ms
^C
--- 10.19.139.27 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.062/0.065/0.072/0.010 ms
(undercloud) [stack@ospha-inst ~]$ netstat -taupen | grep 9000
(No info could be read for "-p": geteuid()=1000 but you should be root.)
tcp        0      0 192.168.9.1:9000        0.0.0.0:*               LISTEN      985        139558     -                   
tcp        0      0 192.168.9.3:9000        0.0.0.0:*               LISTEN      0          127021     -                   
tcp        0      0 10.19.139.27:9000       0.0.0.0:*               LISTEN      0          127020     -                   
tcp       32      0 10.19.139.27:55960      10.19.139.27:13808      CLOSE_WAIT  985        8090004    -                   
(undercloud) [stack@ospha-inst ~]$ systemctl | grep zaqar
  openstack-zaqar@1.service                                                           loaded active running   OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1
  system-openstack\x2dzaqar.slice                                                     loaded active active    system-openstack\x2dzaqar.slice
(undercloud) [stack@ospha-inst ~]$ systemctl status openstack-zaqar@1.service
\u25cf openstack-zaqar@1.service - OpenStack Message Queuing Service (code-named Zaqar) Server Instance 1
   Loaded: loaded (/usr/lib/systemd/system/openstack-zaqar@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-11-02 19:13:58 UTC; 1 day 1h ago
 Main PID: 7887 (zaqar-server)
   CGroup: /system.slice/system-openstack\x2dzaqar.slice/openstack-zaqar@1.service
           \u2514\u25007887 /usr/bin/python2 /usr/bin/zaqar-server --config-file /etc/zaqar/zaqar.conf --config-file /etc/zaqar/1.conf
(undercloud) [stack@ospha-inst ~]$ 


A sosrepot and my config files and deploy command can be found here:

http://refarch.cloud.lab.eng.bos.redhat.com/pub/tmp/bz<#######>



Version-Release number of selected component (if applicable):

This particular kit was pin to 

[root@ospha-inst ~]# grep RH7-RHOS-12.0 /etc/yum.repos.d/rhos-release-12.repo 
# puddle_baseurl=http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/12.0-RHEL-7/2017-10-31.2/RH7-RHOS-12.0/$basearch/os
baseurl=http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/12.0-RHEL-7/2017-10-31.2/RH7-RHOS-12.0/$basearch/os
baseurl=http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/12.0-RHEL-7/latest/RH7-RHOS-12.0/$basearch/debuginfo
baseurl=http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/12.0-RHEL-7/latest/RH7-RHOS-12.0/source
[root@ospha-inst ~]# 


How reproducible:

Each try for the last week, various kits.

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Steve Reichard 2017-11-09 13:27:25 UTC
AS a process of elimination have determined this is related to changes I did in my roles.data

What I used to do here was remove reference to swift, remove Tenant netowrks since using vlan, and remove storage mgmt since there is no swift and I am using external ceph.  Also recently add external to Compute networks since I wanted DVR.

Comment 4 Steve Reichard 2017-11-09 21:24:15 UTC
Closing - user error - my scripts did damage to the roles file.


Note You need to log in before you can comment on or make changes to this bug.