Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1601942 - Unable to delete Heat stacks
Summary: Unable to delete Heat stacks
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat
Version: 11.0 (Ocata)
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Zane Bitter
QA Contact: Ronnie Rasouli
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-17 13:54 UTC by kforde
Modified: 2018-07-26 08:11 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-26 08:11:35 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description kforde 2018-07-17 13:54:36 UTC
Description of problem:

Unable to delete certain heat stacks.
Seems to be related to a recent outage (05/07/18). 

example: 

[stack@undercloud firmware]$ openstack stack delete   f57ad43a-eee2-4b43-aea0-1e1639930135 --wait                                                                                                                                   
Are you sure you want to delete this stack(s) [y/N]? y
2018-07-17 13:49:00Z [baremetal_3237]: DELETE_IN_PROGRESS  Stack DELETE started
2018-07-17 13:49:00Z [baremetal_3237.baremetal_env]: DELETE_IN_PROGRESS  state changed
2018-07-17 13:49:00Z [baremetal_3237.baremetal_env]: DELETE_FAILED  resources.baremetal_env: Timed out
2018-07-17 13:49:00Z [baremetal_3237]: DELETE_FAILED  Resource DELETE failed: resources.baremetal_env: Timed out

 Stack f57ad43a-eee2-4b43-aea0-1e1639930135 DELETE_FAILED 

Unable to delete 1 of the 1 stacks.

[stack@undercloud firmware]$ openstack stack list | grep  f57ad43a-eee2-4b43-aea0-1e1639930135 
| f57ad43a-eee2-4b43-aea0-1e1639930135 | baremetal_3237                                     | DELETE_FAILED      | 2018-07-05T18:40:41Z | 2018-07-17T13:49:00Z |


[stack@undercloud firmware]$ heat resource-list f57ad43a-eee2-4b43-aea0-1e1639930135
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-------------------+--------------------------------------+-------------------------------+-----------------+----------------------+
| resource_name     | physical_resource_id                 | resource_type                 | resource_status | updated_time         |
+-------------------+--------------------------------------+-------------------------------+-----------------+----------------------+
| baremetal_env     | 34ac1b39-f325-4b4f-b52c-324fbfeadb70 | OS::OVB::BaremetalEnvironment | DELETE_FAILED   | 2018-07-05T18:40:42Z |
| private_network   | 1a33ceac-2493-44d3-a57a-86400938ded5 | OS::OVB::PrivateNetwork       | CREATE_COMPLETE | 2018-07-05T18:40:42Z |
| provision_network | c4ee6529-aa32-4860-9805-eed8928e4e82 | OS::Neutron::Net              | CREATE_COMPLETE | 2018-07-05T18:40:42Z |
| provision_subnet  | a7ac12ac-a786-463d-8112-a803475299c5 | OS::Neutron::Subnet           | CREATE_COMPLETE | 2018-07-05T18:40:42Z |
| public_network    | fb94526b-75f5-4bd6-aef9-693213fc4d6f | OS::Neutron::Net              | CREATE_COMPLETE | 2018-07-05T18:40:42Z |
| public_subnet     | e1a0bbdb-79f1-404d-b6bd-1b118ee4635d | OS::Neutron::Subnet           | CREATE_COMPLETE | 2018-07-05T18:40:42Z |
+-------------------+--------------------------------------+-------------------------------+-----------------+----------------------+

[stack@undercloud firmware]$ heat resource-list 34ac1b39-f325-4b4f-b52c-324fbfeadb70
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+--------------------+--------------------------------------+----------------------------+--------------------+----------------------+
| resource_name      | physical_resource_id                 | resource_type              | resource_status    | updated_time         |
+--------------------+--------------------------------------+----------------------------+--------------------+----------------------+
| baremetal_networks | fd1b0c7f-9466-4da9-8e90-a678456b5218 | OS::OVB::BaremetalNetworks | DELETE_IN_PROGRESS | 2018-07-05T18:40:48Z |
+--------------------+--------------------------------------+----------------------------+--------------------+----------------------+


[stack@undercloud firmware]$ heat resource-show  34ac1b39-f325-4b4f-b52c-324fbfeadb70 baremetal_networks                                                                                                                                                                        
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                                                      |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                                                          |
|                        |   "networks": null                                                                                                                                                                                         |
|                        | }                                                                                                                                                                                                          |
| creation_time          | 2018-07-05T18:40:48Z                                                                                                                                                                                       |
| description            |                                                                                                                                                                                                            |
| links                  | https://XXXX:13004/v1/5d8fad99d6e14040b794ea2b0c4d91f5/stacks/baremetal_3237-baremetal_env-hxl7zr7knbqg/34ac1b39-f325-4b4f-b52c-324fbfeadb70/resources/baremetal_networks (self)      |
|                        | https://XXXX:13004/v1/5d8fad99d6e14040b794ea2b0c4d91f5/stacks/baremetal_3237-baremetal_env-hxl7zr7knbqg/34ac1b39-f325-4b4f-b52c-324fbfeadb70 (stack)                                  |
|                        | https://XXXX:13004/v1/8dbc460fe5fd4fcab0096c2c0aad3ece/stacks/baremetal_3237-baremetal_env-hxl7zr7knbqg-baremetal_networks-hfwuhyvpq4z7/fd1b0c7f-9466-4da9-8e90-a678456b5218 (nested) |
| logical_resource_id    | baremetal_networks                                                                                                                                                                                         |
| parent_resource        | baremetal_env                                                                                                                                                                                              |
| physical_resource_id   | fd1b0c7f-9466-4da9-8e90-a678456b5218                                                                                                                                                                       |
| required_by            |                                                                                                                                                                                                            |
| resource_name          | baremetal_networks                                                                                                                                                                                         |
| resource_status        | DELETE_IN_PROGRESS                                                                                                                                                                                         |
| resource_status_reason | state changed                                                                                                                                                                                              |
| resource_type          | OS::OVB::BaremetalNetworks                                                                                                                                                                                 |
| updated_time           | 2018-07-05T18:40:48Z                                                                                                                                                                                       |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+



Version-Release number of selected component (if applicable):
[root@overcloud-controller-2 heat]# rpm -qa | grep heat
heat-cfntools-1.3.0-2.el7.noarch
python-heat-agent-1.0.1-1.el7.noarch
openstack-heat-api-cfn-8.0.7-0.20180418100532.b9a2f48.el7.centos.noarch
python-heat-agent-hiera-1.0.1-1.el7.noarch
openstack-heat-api-cloudwatch-8.0.7-0.20180418100532.b9a2f48.el7.centos.noarch
python-heat-agent-puppet-1.0.1-1.el7.noarch
openstack-heat-agents-1.0.1-1.el7.noarch
python2-heatclient-1.8.2-1.el7.noarch
openstack-heat-common-8.0.7-0.20180418100532.b9a2f48.el7.centos.noarch
openstack-heat-api-8.0.7-0.20180418100532.b9a2f48.el7.centos.noarch
python-heat-agent-apply-config-1.0.1-1.el7.noarch
puppet-heat-10.4.1-0.20180213200235.cdd2cd2.el7.centos.noarch
python-heat-agent-docker-cmd-1.0.1-1.el7.noarch
openstack-heat-engine-8.0.7-0.20180418100532.b9a2f48.el7.centos.noarch
python-heat-agent-json-file-1.0.1-1.el7.noarch
python-heat-agent-ansible-1.0.1-1.el7.noarch


How reproducible:

Not sure how it got in this state

Steps to Reproduce:
1. ??
2.
3.

Actual results:

Stack fails to delete ... see example above

Expected results:

Stack deletes all resources and itself. 

Additional info:

We have several stacks in this state.

Comment 1 Thomas Hervé 2018-07-19 14:21:09 UTC
Can you provide heat engines logs from the failure time ? Thanks.

Comment 2 Zane Bitter 2018-07-19 19:18:10 UTC
The delete appears to be failing immediately, so it looks like this is due to a known race condition with nested stacks, where Heat can see the *previous* DELETE_FAILED status and assume that it refers to the current delete, which hasn't started yet:

https://storyboard.openstack.org/#!/story/1669608

In theory the underlying nested stack should actually get deleted (the resource in it shows as DELETE_IN_PROGRESS in your output) anyway, after which a subsequent delete should actually remove the parent resource.

Failing that, a workaround is to get the resource into some other state, which will avoid the race condition. You can do that using the "openstack stack check" command (that should put it into a CHECK_FAILED state).

It's likely the initial failure was related to the outage you mentioned.

If none of that works then we'll need to see logs to figure out what has happened.

Comment 3 kforde 2018-07-20 09:52:58 UTC
Thanks Zane for update on that race condition. 

So, we recently did a minor update of the Overcloud. 
These stacks are now deleted and showing DELETE_COMPLETE. 

I didn't take any action here. 

Regarding the suggestion of changing the 'state', I did that as a first option.
The 'state' changed to CHECK_COMPLETE and not CHECK_FAILED and the deletion failed immediately. 

For now, I think we can close this issue.


Note You need to log in before you can comment on or make changes to this bug.