Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1358491 - Overcloud deploy with Cinder failed on Resource CREATE failed: Error: resources.BlockStorageNodesPostDeployment.resources.VolumeDeployment_Step2
Summary: Overcloud deploy with Cinder failed on Resource CREATE failed: Error: resourc...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 10.0 (Newton)
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: ---
: 10.0 (Newton)
Assignee: Angus Thomas
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-20 20:12 UTC by Dan Yasny
Modified: 2016-10-14 15:34 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-14 15:34:49 UTC


Attachments (Terms of Use)

Description Dan Yasny 2016-07-20 20:12:27 UTC
Description of problem:
 Deployment command: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --swift-storage-scale 1 --block-storage-scale 1 --neutron-tunnel-types vxlan,gre --neutron-network-type vxlan,gre --neutron-network-vlan-ranges datacentre:118:143 --neutron-bridge-mappings datacentre:br-ex  --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

The deployment fails on block storage:
[stack@instack ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+---------------+---------------------+--------------+
| id                                   | stack_name | stack_status  | creation_time       | updated_time |
+--------------------------------------+------------+---------------+---------------------+--------------+
| c1a0f40c-f357-42c8-b1d7-702ba94509a7 | overcloud  | CREATE_FAILED | 2016-07-20T18:19:20 | None         |
+--------------------------------------+------------+---------------+---------------------+--------------+


[stack@instack ~]$ heat stack-list --show-nested -f "status=FAILED"
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+--------------------------------------------------------------------------------------------+---------------+---------------------+--------------+--------------------------------------+
| id                                   | stack_name                                                                                 | stack_status  | creation_time       | updated_time | parent                               |
+--------------------------------------+--------------------------------------------------------------------------------------------+---------------+---------------------+--------------+--------------------------------------+
| c1a0f40c-f357-42c8-b1d7-702ba94509a7 | overcloud                                                                                  | CREATE_FAILED | 2016-07-20T18:19:20 | None         | None                                 |
| 56dd9e0c-dc22-47c7-a1c6-7bf658f61fec | overcloud-BlockStorageNodesPostDeployment-qh6kejxg3cp3                                     | CREATE_FAILED | 2016-07-20T19:17:02 | None         | c1a0f40c-f357-42c8-b1d7-702ba94509a7 |
| 570aea1e-ff8f-4300-ac16-6ea467f83866 | overcloud-BlockStorageNodesPostDeployment-qh6kejxg3cp3-VolumeDeployment_Step2-kish2cvl6hpg | CREATE_FAILED | 2016-07-20T19:17:50 | None         | 56dd9e0c-dc22-47c7-a1c6-7bf658f61fec |
+--------------------------------------+--------------------------------------------------------------------------------------------+---------------+---------------------+--------------+--------------------------------------+


[stack@instack ~]$ heat resource-list --nested-depth 5 overcloud | grep FAILED
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
| BlockStorageNodesPostDeployment               | 56dd9e0c-dc22-47c7-a1c6-7bf658f61fec          | OS::TripleO::BlockStoragePostDeployment                                                       | CREATE_FAILED   | 2016-07-20T18:19:20 | overcloud                                                                                                                                         |
| VolumeDeployment_Step2                        | 570aea1e-ff8f-4300-ac16-6ea467f83866          | OS::Heat::StructuredDeployments                                                               | CREATE_FAILED   | 2016-07-20T19:17:03 | overcloud-BlockStorageNodesPostDeployment-qh6kejxg3cp3                                                                                            |
| 0                                             | 44f72229-1c31-405d-bbe1-905ce0c72662          | OS::Heat::StructuredDeployment                                                                | CREATE_FAILED   | 2016-07-20T19:17:50 | overcloud-BlockStorageNodesPostDeployment-qh6kejxg3cp3-VolumeDeployment_Step2-kish2cvl6hpg                                                        |



[stack@instack ~]$ heat resource-show 570aea1e-ff8f-4300-ac16-6ea467f83866 0
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                                                               |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                                                                   |
|                        |   "deploy_stdout": null,                                                                                                                                                                                            |
|                        |   "deploy_stderr": null,                                                                                                                                                                                            |
|                        |   "deploy_status_code": null                                                                                                                                                                                        |
|                        | }                                                                                                                                                                                                                   |
| creation_time          | 2016-07-20T19:17:50                                                                                                                                                                                                 |
| description            |                                                                                                                                                                                                                     |
| links                  | http://192.0.2.1:8004/v1/5c89ca4d3de44a31935c8dcc05cdb6a4/stacks/overcloud-BlockStorageNodesPostDeployment-qh6kejxg3cp3-VolumeDeployment_Step2-kish2cvl6hpg/570aea1e-ff8f-4300-ac16-6ea467f83866/resources/0 (self) |
|                        | http://192.0.2.1:8004/v1/5c89ca4d3de44a31935c8dcc05cdb6a4/stacks/overcloud-BlockStorageNodesPostDeployment-qh6kejxg3cp3-VolumeDeployment_Step2-kish2cvl6hpg/570aea1e-ff8f-4300-ac16-6ea467f83866 (stack)            |
| logical_resource_id    | 0                                                                                                                                                                                                                   |
| parent_resource        | VolumeDeployment_Step2                                                                                                                                                                                              |
| physical_resource_id   | 44f72229-1c31-405d-bbe1-905ce0c72662                                                                                                                                                                                |
| required_by            |                                                                                                                                                                                                                     |
| resource_name          | 0                                                                                                                                                                                                                   |
| resource_status        | CREATE_FAILED                                                                                                                                                                                                       |
| resource_status_reason | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1                                                                                               |
| resource_type          | OS::Heat::StructuredDeployment                                                                                                                                                                                      |
| updated_time           | 2016-07-20T19:17:50                                                                                                                                                                                                 |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+



WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED", 
  "server_id": "87903124-f19c-4af4-9941-64aa331abcdf", 
  "config_id": "20b12cc2-b0f7-4599-9f19-19c49c733392", 
  "output_values": {
    "deploy_stdout": "", 
    "deploy_stderr": "\u001b[1;31mError: Could not find data item bootstrap_nodeid in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/pacemaker/cinder/volume.pp:31 on node overcloud-blockstorage-0.localdomain\u001b[0m\n\u001b[1;31mError: Could not find data item bootstrap_nodeid in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/pacemaker/cinder/volume.pp:31 on node overcloud-blockstorage-0.localdomain\u001b[0m\n", 
    "deploy_status_code": 1
  }, 
  "creation_time": "2016-07-20T19:17:51", 
  "updated_time": "2016-07-20T19:18:19", 
  "input_values": {
    "step": 2, 
    "update_identifier": "1469038752"
  }, 
  "action": "CREATE", 
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", 
  "id": "44f72229-1c31-405d-bbe1-905ce0c72662"
}


Version-Release number of selected component (if applicable):
openstack-tripleo-puppet-elements-5.0.0-0.20160630200107.e2ec53e.el7ost.noarch
openstack-neutron-9.0.0-0.20160706092201.0f8e23f.el7ost.noarch
openstack-nova-cert-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160705210328.0fbe987.el7ost.noarch
openstack-neutron-ml2-9.0.0-0.20160706092201.0f8e23f.el7ost.noarch
openstack-tripleo-image-elements-5.0.0-0.20160621221252.91128c1.el7ost.noarch
openstack-heat-api-7.0.0-0.20160706042842.5dc4d26.el7ost.noarch
openstack-swift-account-2.8.1-0.20160705194315.79be80f.el7ost.noarch
openstack-sahara-engine-5.0.0-0.20160706121028.439a5a4.el7ost.noarch
openstack-mistral-engine-3.0.0-0.20160706132830.5437ad4.el7ost.noarch
puppet-openstacklib-9.0.0-0.20160701002107.413a614.el7ost.noarch
openstack-zaqar-3.0.0-0.20160706062751.6e0e881.el7ost.noarch
openstack-ironic-api-6.0.1-0.20160705221003.9322861.el7ost.noarch
openstack-nova-common-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-swift-proxy-2.8.1-0.20160705194315.79be80f.el7ost.noarch
openstack-mistral-common-3.0.0-0.20160706132830.5437ad4.el7ost.noarch
openstack-heat-common-7.0.0-0.20160706042842.5dc4d26.el7ost.noarch
openstack-heat-templates-0.0.1-0.20160614213831.b9f87c5.el7ost.noarch
openstack-nova-cells-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-nova-console-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-nova-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-heat-engine-7.0.0-0.20160706042842.5dc4d26.el7ost.noarch
openstack-tripleo-common-5.0.0-0.20160705162843.7f2cefe.el7ost.noarch
python-openstack-mistral-3.0.0-0.20160706132830.5437ad4.el7ost.noarch
openstack-neutron-common-9.0.0-0.20160706092201.0f8e23f.el7ost.noarch
openstack-ironic-common-6.0.1-0.20160705221003.9322861.el7ost.noarch
puppet-openstack_extras-9.0.0-0.20160705205608.7c15f30.el7ost.noarch
python-openstackclient-2.6.1-0.20160706112804.1b05a6d.el7ost.noarch
openstack-sahara-common-5.0.0-0.20160706121028.439a5a4.el7ost.noarch
openstack-nova-novncproxy-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-nova-api-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-mistral-api-3.0.0-0.20160706132830.5437ad4.el7ost.noarch
openstack-ironic-conductor-6.0.1-0.20160705221003.9322861.el7ost.noarch
openstack-neutron-openvswitch-9.0.0-0.20160706092201.0f8e23f.el7ost.noarch
openstack-mistral-executor-3.0.0-0.20160706132830.5437ad4.el7ost.noarch
openstack-swift-container-2.8.1-0.20160705194315.79be80f.el7ost.noarch
openstack-sahara-5.0.0-0.20160706121028.439a5a4.el7ost.noarch
openstack-nova-scheduler-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-keystone-10.0.0-0.20160706110212.a6af6b4.el7ost.noarch
openstack-tempest-12.0.1-0.20160622103404.6e60cd5.1.el7ost.noarch
python-openstacksdk-0.8.7-0.20160705213311.ccd8daa.el7ost.noarch
openstack-nova-compute-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-tripleo-0.0.1-0.20160622171438.99ccd66.el7ost.noarch
openstack-glance-13.0.0-0.20160706012220.8abaa5a.el7ost.noarch
openstack-nova-network-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-heat-api-cfn-7.0.0-0.20160706042842.5dc4d26.el7ost.noarch
openstack-sahara-api-5.0.0-0.20160706121028.439a5a4.el7ost.noarch
openstack-nova-conductor-14.0.0-0.20160706131636.1f4711a.1.el7ost.noarch
openstack-swift-2.8.1-0.20160705194315.79be80f.el7ost.noarch
openstack-puppet-modules-9.0.0-0.20160630235319.8c758d6.el7ost.noarch
openstack-cinder-9.0.0-0.20160706082144.afb81cd.el7ost.noarch
openstack-ironic-inspector-3.3.1-0.20160704180215.9265a61.el7ost.noarch
openstack-swift-object-2.8.1-0.20160705194315.79be80f.el7ost.noarch
openstack-selinux-0.7.3-3.el7ost.noarch

[stack@instack ~]$ rhos-release -v
1.0.49


[stack@instack ~]$ yum repolist
Loaded plugins: search-disabled-repos
rhelosp-10.0-ceph-1.3-mon                                                                                                                                                                                                               26/26
rhelosp-10.0-ceph-1.3-osd                                                                                                                                                                                                               23/23
rhelosp-10.0-puddle                                                                                                                                                                                                                   653/653
repo id                                                                                                  repo name                                                                                                                      status
rhelosp-10.0-ceph-1.3-mon                                                                                Ceph 1.3 MON                                                                                                                      26
rhelosp-10.0-ceph-1.3-osd                                                                                Ceph 1.3 OSD                                                                                                                      23
rhelosp-10.0-puddle/x86_64                                                                               RHOS-10.0                                                                                                                        653
rhelosp-rhel-7-common/x86_64                                                                             RHEL 7 Common                                                                                                                    196
rhelosp-rhel-7.2-extras/x86_64                                                                           Red Hat Enterprise Linux 7Server - x86_64 - Extras                                                                                85
rhelosp-rhel-7.2-ha/x86_64                                                                               Red Hat Enterprise Linux 7Server - x86_64 - HA                                                                                    30
rhelosp-rhel-7.2-server/x86_64                                                                           Red Hat Enterprise Linux 7Server - x86_64 - Server                                                                             4,620
rhelosp-rhel-7.2-z/x86_64                                                                                Red Hat Enterprise Linux 7Server - x86_64 - Z-Stream                                                                           8,579
rhos-release                                                                                             RHOS Release                                                                                                                     112
rhos-release-extras/7Server                                                                              RHOS Release Extras                                                                                                                2



Steps to Reproduce:
 see deploy command and used yum repos above

Actual results:
deployment failed

Expected results:
deployment succeeded

Comment 2 Jaromir Coufal 2016-10-11 03:55:49 UTC
Dan is this reproducable?

Comment 3 Dan Yasny 2016-10-11 13:24:02 UTC
Just started a new deployment trying to reproduce:
 Deployment command: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --swift-storage-scale 1 --block-storage-scale 1 --neutron-tunnel-types vxlan,gre --neutron-network-type vxlan,gre --neutron-network-vlan-ranges datacentre:118:143 --neutron-bridge-mappings datacentre:br-ex  --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

Will update here once it's done

Comment 4 Dan Yasny 2016-10-11 15:40:15 UTC
The deployment failed with a different error this time:

https://bugzilla.redhat.com/show_bug.cgi?id=1383730

Comment 5 James Slagle 2016-10-14 15:34:49 UTC
alright closing this one out then, please reopen if the original error reproduces


Note You need to log in before you can comment on or make changes to this bug.