Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1361795 - rhel-osp-director: 7.2->7.3Async->8.0 fails: Error: unable to push cib Call cib_replace failed (-205): Update was older than existing configuration
Summary: rhel-osp-director: 7.2->7.3Async->8.0 fails: Error: unable to push cib Call...
Keywords:
Status: CLOSED DUPLICATE of bug 1361746
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: async
: 8.0 (Liberty)
Assignee: Angus Thomas
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-30 15:15 UTC by Alexander Chuzhoy
Modified: 2017-02-09 17:34 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-09 17:34:04 UTC


Attachments (Terms of Use)

Description Alexander Chuzhoy 2016-07-30 15:15:08 UTC
rhel-osp-director:  7.2->7.3Async->8.0 fails: Error: unable to push cib  Call cib_replace failed (-205): Update was older than existing configuration

Environment:
openstack-tripleo-heat-templates-kilo-0.8.14-16.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch
instack-undercloud-2.2.7-7.el7ost.noarch
openstack-puppet-modules-7.1.3-1.el7ost.noarch

Steps to reproduce:
1. deploy 7.2 with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

2. Populate the overcloud
3. Update the setup to 7.3async
4. Attempt to upgrade the setup to 8.0
Result:
The upgrade fails during the major-upgrade-pacemaker-init.yaml step:

2016-07-30 05:14:07 [ControllerConfig]: CREATE_IN_PROGRESS  state changed
2016-07-30 05:14:09 [ControllerConfig]: CREATE_COMPLETE  state changed
2016-07-30 05:14:10 [NodeTLSData]: UPDATE_COMPLETE  state changed
2016-07-30 05:14:11 [ControllerConfig]: UPDATE_IN_PROGRESS  state changed
2016-07-30 05:14:11 [ControllerConfig]: CREATE_IN_PROGRESS  state changed
2016-07-30 05:14:12 [ControllerConfig]: CREATE_COMPLETE  state changed
ERROR: Authentication failed: Authentication required


[stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+----------------------------------------------------------------------------------+
| resource_name                             | physical_resource_id                          | resource_type                                     | resource_status | updated_time        | stack_name                                                                       |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+----------------------------------------------------------------------------------+
| CephStorage                               | 855b0fea-0ca4-48b3-ba42-28c6c137369a          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2016-07-30T05:11:24 | overcloud                                                                        |
| 0                                         | 117b0cbe-8086-4bb0-8502-4105c37b5986          | OS::TripleO::CephStorage                          | UPDATE_FAILED   | 2016-07-30T05:11:26 | overcloud-CephStorage-kalfd7gcktuj                                               |
| Compute                                   | 99827da2-4942-4638-ba3e-8fb87ffe0d92          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2016-07-30T05:11:46 | overcloud                                                                        |
| 0                                         | 81ae7c82-376e-4815-b558-2f28b9b38ba7          | OS::TripleO::Compute                              | UPDATE_FAILED   | 2016-07-30T05:11:49 | overcloud-Compute-s57fmy7wi4pq                                                   |
| Controller                                | e5393d15-bee5-4793-ac53-3cf05cff9818          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2016-07-30T05:11:51 | overcloud                                                                        |
| 1                                         | 690c706a-730e-4320-a2b1-b15f54f64ff8          | OS::TripleO::Controller                           | UPDATE_FAILED   | 2016-07-30T05:11:59 | overcloud-Controller-oewxp33qsmih                                                |
| UpdateDeployment                          | 0ab759e2-8bed-48ce-9548-1d9693a8c8df          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:12:00 | overcloud-CephStorage-kalfd7gcktuj-0-z3cpkaozmlf5                                |
| 2                                         | 082e9ec1-1b37-4ed6-94ad-ee20f253f5d0          | OS::TripleO::Controller                           | UPDATE_FAILED   | 2016-07-30T05:12:03 | overcloud-Controller-oewxp33qsmih                                                |
| 0                                         | 6b5be9dd-22c3-4bb2-9685-610691cb5e8a          | OS::TripleO::Controller                           | UPDATE_FAILED   | 2016-07-30T05:12:09 | overcloud-Controller-oewxp33qsmih                                                |
| UpdateDeployment                          | 60eb88da-685c-4b57-9c98-8dc8d409c493          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:12:26 | overcloud-Compute-s57fmy7wi4pq-0-cze7ikb45td6                                    |
| UpdateDeployment                          | 79b712d2-b329-4872-aa5d-3d26f9ba103a          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:12:54 | overcloud-Controller-oewxp33qsmih-1-tvyjay4p73wa                                 |
| UpdateDeployment                          | b30ea3b6-5ef1-418e-ad9c-51aac8b57372          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:13:13 | overcloud-Controller-oewxp33qsmih-0-okt4h7ois7wo                                 |
| UpdateDeployment                          | a578aa2e-6089-4b1d-ad72-df8677681d33          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:13:20 | overcloud-Controller-oewxp33qsmih-2-7ho5f2fms7wi                                 |
+-------------------------------------------+---------------


[stack@instack ~]$ echo -e `heat deployment-show 79b712d2-b329-4872-aa5d-3d26f9ba103a`|grep -i -A2 -B2 error
+ echo 'Applying new Pacemaker config'
+ pcs cluster cib-push /tmp/tmp.CBVk9DMtTj
Error: unable to push cib
Call cib_replace failed (-205): Update was older than existing configuration


No stopped pacemaker resources.

Comment 3 Marios Andreou 2016-08-01 08:33:45 UTC
Hi sasha o/ I left a comment at the other bug you mentioned that you were chasing when you hit the bug you report here, at https://bugzilla.redhat.com/show_bug.cgi?id=1361746#c3

I think it may also apply here as the root cause, in particular the "ERROR: Authentication failed: Authentication required"

Comment 4 Jiri Stransky 2016-08-01 09:41:13 UTC
(In reply to Alexander Chuzhoy from comment #0)
> [stack@instack ~]$ echo -e `heat deployment-show
> 79b712d2-b329-4872-aa5d-3d26f9ba103a`|grep -i -A2 -B2 error
> + echo 'Applying new Pacemaker config'
> + pcs cluster cib-push /tmp/tmp.CBVk9DMtTj
> Error: unable to push cib
> Call cib_replace failed (-205): Update was older than existing configuration

This comes from yum_update.sh, which should be no-opped at the time of upgrade init:

[stack@instack ~]$ grep PackageUpdate /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml
  OS::TripleO::Tasks::PackageUpdate:  ../extraconfig/tasks/yum_update_noop.yaml

Perhaps the error with Keystone trusts that Marios linked above could be the root cause of the cib_replace error too (maybe yum_update.sh wasn't correctly no-opped because of that error?).

Comment 5 Sofer Athlan-Guyot 2017-02-09 17:34:04 UTC
Hi Sasha,

closing it as very old.  Don't hesitate to re-open it if the _member_ doc tricks wasn't enough to fix this one.

*** This bug has been marked as a duplicate of bug 1361746 ***


Note You need to log in before you can comment on or make changes to this bug.