Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1596866 - Heat database error: 'Cannot delete or update a parent row: a foreign key constraint fails ...
Summary: Heat database error: 'Cannot delete or update a parent row: a foreign key con...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z2
: 13.0 (Queens)
Assignee: Zane Bitter
QA Contact: Ronnie Rasouli
URL:
Whiteboard:
Depends On: 1577874
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-29 20:22 UTC by Zane Bitter
Modified: 2018-08-29 16:21 UTC (History)
12 users (show)

Fixed In Version: openstack-heat-10.0.1-2.el7ost
Doc Type: Bug Fix
Doc Text:
When a stack has more than 1,000 past events associated with it, Heat begins to purge a portion of existing events from the database, with some probability of this occurring each time a new event is created. However, if the stack has had previous updates with convergence disabled (the default on the overcloud prior to OSP10, and on the undercloud prior to OSP13) some events might reference resource properties data from the backup stack. Purging events may fail with a foreign key constraint error: 'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))' or 'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`resource`, CONSTRAINT `rsrc_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))' This prevents the new event from being stored and the stack update fails as a result. Heat now ignores foreign key constraint errors when attempting to purge events. If this situation occurs, events are not purged until any backup stacks that hold common references have been purged. New events are stored and the operation of the stack continues.
Clone Of: 1577874
Environment:
Last Closed: 2018-08-29 16:21:37 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack Storyboard 2002643 None None None 2018-06-29 20:22:33 UTC
OpenStack gerrit 578816 None stable/queens: MERGED heat: Ignore errors in purging events (Iefa706f91382f7d1c47e09e8f67a9fad53a9390b) 2018-07-19 02:10:37 UTC
Red Hat Product Errata RHBA-2018:2590 None None None 2018-08-29 16:21:42 UTC

Description Zane Bitter 2018-06-29 20:22:33 UTC
+++ This bug was initially created as a clone of Bug #1577874 +++

Description of problem:

This error can be seen in a OSP12 deployment:

2018-05-14 04:38:02Z [overcloud-Controller-irffjxb5zlgh-2-u56vlld5py3g]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-05-14 04:38:03Z [overcloud-Controller-irffjxb5zlgh.2]: UPDATE_COMPLETE  state changed
2018-05-14 04:38:03Z [overcloud-Controller-irffjxb5zlgh]: UPDATE_FAILED  resources[0]: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') 
2018-05-14 04:38:04Z [Controller]: UPDATE_FAILED  resources.Controller: resources[0]: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_prope
2018-05-14 04:42:05Z [Compute]: UPDATE_FAILED  UPDATE aborted
2018-05-14 04:42:05Z [overcloud]: UPDATE_FAILED  resources.Controller: resources[0]: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_prope

 Stack overcloud UPDATE_FAILED 

overcloud.Controller.0:
  resource_type: OS::TripleO::Controller
  physical_resource_id: 7173f444-fe34-417d-bfeb-c853ee85bc06
  status: UPDATE_FAILED
  status_reason: |
    resources[0]: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s, %(id_6)s, %(id_7)s, %(id_8)s, %(id_9)s, %(id_10)s)'] [parameters: {u'id_10': 6781, u'id_2': 6785, u'id_3': 7014, u'id_1': 7008, u'id_6': 6962, u'id_7': 6964, u'id_4': 6731, u'id_5': 6736, u'id_8': 6871, u'id_9': 6868}]
overcloud.Compute:
  resource_type: OS::Heat::ResourceGroup
  physical_resource_id: cb7ed374-46dd-4121-a9a5-e982100e3123
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted


It seems like the following upstream bug [1] solved in penstack/heat 9.0.0.0b2    but seen in  openstack-heat-engine-9.0.1-3

[1] https://bugs.launchpad.net/heat/+bug/1681772

Version-Release number of selected component (if applicable):


How reproducible:
Unsure


Steps to Reproduce:
1. Scale up the overcloud
2.
3.

Actual results:
Database error

Expected results:
no database error

Additional info:

--- Additional comment from Zane Bitter on 2018-06-22 23:04:24 EDT ---

I found the root cause:

https://storyboard.openstack.org/#!/story/2002643#comment-90440

This affects all non-convergence stacks in OSP11 and up (although in practice that only means TripleO).

--- Additional comment from Zane Bitter on 2018-06-29 16:19:20 EDT ---

We're in the process of backporting a patch to ignore (but log) the error upstream.

The root cause is that entries in the resource properties data can be shared between events/resources in the current stack and those in the backup stack (for non-covergence stacks, including TripleO prior to OSP13). This will be fully fixed on master.

If anybody encounters this in the meantime, a good workaround to try would be to purge deleted stacks using 'heat-manage purge_deleted', since the unwanted references may be in now-deleted backup stacks. There's no guarantee, however, since the references may in fact be from a current backup stack.

Comment 8 Ronnie Rasouli 2018-08-22 07:25:42 UTC
modified the config.py as

https://review.openstack.org/#/c/577432/13/heat/common/config.py

Scaled out compute node successfully

Comment 10 errata-xmlrpc 2018-08-29 16:21:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2590


Note You need to log in before you can comment on or make changes to this bug.