Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1364240 - RHOSP Deployment dies at 30%
Summary: RHOSP Deployment dies at 30%
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Red Hat Quickstart Cloud Installer
Classification: Red Hat
Component: Installation - RHELOSP
Version: 1.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ga
: 1.0
Assignee: John Matthews
QA Contact: Dave Johnson
Julie
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-04 19:04 UTC by Thom Carlin
Modified: 2016-08-12 13:46 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-12 13:41:25 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1347063 None None None Never
Red Hat Bugzilla 1358779 None None None Never

Internal Links: 1347063 1358779

Description Thom Carlin 2016-08-04 19:04:33 UTC
Description of problem:

RHOSP deployment errors at 30%

Version-Release number of selected component (if applicable):

QCI-1.0-RHEL-7-20160801.t.2
QCIOOO-8.0-RHEL-7-20160801.t.0

How reproducible:

Uncertain

Steps to Reproduce:
1. Install/configure QCI/TripleO
2. Deploy RHOSP in error
3. Remove errant deployment
4. Deploy RHOSP again

Actual results:

Error at 30%
Failed deployment

Expected results:

Successful deployment

Additional info:

Actions::Fusor::Deployment::OpenStack::Deploy
Deploy Red Hat RHEL OpenStack Platform Platform overcloud

ERROR: deployment failed with status: CREATE_FAILED and reason: Resource CREATE failed: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned

Comment 1 Thom Carlin 2016-08-04 19:17:37 UTC
heat-api[10150]: 2016-08-05 02:56:13.969 10150 DEBUG heat.common.serializers [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] JSON response : {"stacks": [{"description": "Deploy an OpenStack environment, consisting of several node types (roles), Controller, Compute, BlockStorage, SwiftStorage and CephStorage.  The Storage roles enable independent scaling of the storage components, but the minimal deployment is one Controller and one Compute node.\n", "parent": null, "stack_status_reason": "Resource CREATE failed: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned", "stack_name": "overcloud", "stack_user_project_id": "960fbf8186e840adaf70d545094024cc", "tags": null, "creation_time": "2016-08-05T06:53:08", "links": [{"href": "http://<<undercloud_ip_address>>:8004/v1/53a34f2f0360432b9b6c58e1c0a15c33/stacks/overcloud/bc9082b7-454e-4711-b9a0-1a5e5621845d", "rel": "self"}], "updated_time": null, "stack_owner": "admin", "stack_status": "CREATE_FAILED", "id": "bc9082b7-454e-4711-b9a0-1a5e5621845d"}]} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42

heat-engine: 2016-08-05 02:53:34.204 10258 DEBUG dogpile.core.dogpile [req-a4a24a7c-2cbe-47c8-ab83-3475ba332b31 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154
heat-engine: 2016-08-05 02:53:34.205 10258 DEBUG heat.engine.stack [req-a4a24a7c-2cbe-47c8-ab83-3475ba332b31 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned validate /usr/lib/python2.7/site-packages/heat/engine/stack.py:678
heat-engine: 2016-08-05 02:53:34.206 10258 DEBUG oslo_messaging.rpc.dispatcher [req-a4a24a7c-2cbe-47c8-ab83-3475ba332b31 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] Expected exception during message handling (Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned) _dispatch_and_reply /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:145
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource [-] update_stack
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource Traceback (most recent call last):
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 434, in update_with_template
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     args)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 269, in update_stack
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     args=args))
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 61, in call
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     return client.call(ctxt, method, **kwargs)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 403, in call
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     return self.prepare().call(ctxt, method, **kwargs)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     retry=self.retry)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     timeout=timeout, retry=retry)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     retry=retry)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     raise result
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource StackValidationFailed_Remote: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource
heat-engine: 2016-08-05 02:53:34.287 10248 INFO heat.engine.resource [-] CREATE: ResourceGroup "Controller" [79e92a81-4740-4a4b-9398-fa6541cb3f3e] Stack "overcloud" [bc9082b7-454e-4711-b9a0-1a5e5621845d]
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource Traceback (most recent call last):
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     yield
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 708, in _do_action
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 292, in wrapper
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     step = next(subtask)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 679, in action_handler_task
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     handler_data = handler(*args)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 370, in handle_create
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     checkers[0].start()
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 194, in start
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     self.step()
:
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 217, in step
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     next(self._runner)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 385, in _run_to_completion
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     timeout)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 437, in update_with_template
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     self.raise_local_exception(ex)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 328, in raise_local_exception
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     raise exception.ResourceFailure(message, self, action=self.action)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource ResourceFailure: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource

Comment 2 Thom Carlin 2016-08-04 19:21:15 UTC
2016-08-05 02:56:13.923 10150 DEBUG oslo_messaging._drivers.amqpdriver [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] MSG_ID is bfa8225b5b
9046ccac20d94b1f04d580 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-08-05 02:56:13.969 10150 DEBUG heat.common.serializers [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] JSON response : {"stacks": [{"d
escription": "Deploy an OpenStack environment, consisting of several node types (roles), Controller, Compute, BlockStorage, SwiftStorage and CephStorage.  The Storage roles enable independent scaling 
of the storage components, but the minimal deployment is one Controller and one Compute node.\n", "parent": null, "stack_status_reason": "Resource CREATE failed: resources.Controller: Property error: 
resources[0].properties: Property MysqlClustercheckPassword not assigned", "stack_name": "overcloud", "stack_user_project_id": "960fbf8186e840adaf70d545094024cc", "tags": null, "creation_time": "2016-
08-05T06:53:08", "links": [{"href": "http://<<undercloud_ip_address>>:8004/v1/53a34f2f0360432b9b6c58e1c0a15c33/stacks/overcloud/bc9082b7-454e-4711-b9a0-1a5e5621845d", "rel": "self"}], "updated_time": null, "stack
_owner": "admin", "stack_status": "CREATE_FAILED", "id": "bc9082b7-454e-4711-b9a0-1a5e5621845d"}]} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42
2016-08-05 02:56:13.970 10150 DEBUG eventlet.wsgi.server [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] <<gateway>> - - [05/Aug/2016 0
2:56:13] "GET /v1/53a34f2f0360432b9b6c58e1c0a15c33/stacks HTTP/1.1" 200 1103 0.125543 write /usr/lib/python2.7/site-packages/heat/common/wsgi.py:267

Comment 3 Thom Carlin 2016-08-04 19:29:41 UTC
Resume results in "ERROR: deployment failed with status: UPDATE_IN_PROGRESS and reason: Stack UPDATE started"

Comment 4 Thom Carlin 2016-08-04 20:44:28 UTC
CREATED_FAILED, UPDATE_FAILED on compute node:
resources.Compute: resources[0]: ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"

Comment 5 Thom Carlin 2016-08-05 19:08:37 UTC
Possibly bz 1358779 (Testing https://review.openstack.org/#/c/337304/ against /usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml /home/stack/openstack-tripleo-heat-templates/puppet/controller.yaml

Comment 7 Jason Montleon 2016-08-12 13:34:00 UTC
Known issue with Openstack it appears:
https://access.redhat.com/solutions/2472361

Comment 8 Jason Montleon 2016-08-12 13:41:25 UTC
There's nothing much we can do for this. Either don't do a yum update or keep these packages at the versions listed until a newer version resolves the issue.


Note You need to log in before you can comment on or make changes to this bug.