Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1354046

Summary: rhel-osp-director: 8.0->9.0 upgrade, Keystone WSGI migration step fails, httpd resource is down on the cluster and fails to start.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: Michele Baldessari <michele>
Status: CLOSED ERRATA QA Contact: Rodrigo Duarte <rduartes>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: dbecker, dcritch, jason.dobies, jjoyce, kbasil, mburns, michele, morazi, nlevinki, rcritten, rduartes, rhel-osp-director-maint, sasha, sgordon, tvignaud
Target Milestone: gaKeywords: Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-2.0.0-21.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 11:35:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Alexander Chuzhoy 2016-07-08 20:05:16 UTC
rhel-osp-director:  8.0->9.0 upgrade, Keystone WSGI migration step fails, httpd resource is down on the cluster and fails to start.


Environment:
httpd-tools-2.4.6-40.el7_2.1.x86_64
python-keystoneclient-1.7.2-1.el7ost.noarch
python-keystoneauth1-1.1.0-4.el7ost.noarch
python-keystone-8.0.1-1.el7ost.noarch
httpd-2.4.6-40.el7_2.1.x86_64
python-keystonemiddleware-2.3.1-1.el7ost.noarch
openstack-keystone-8.0.1-1.el7ost.noarch
openstack-tripleo-heat-templates-kilo-2.0.0-13.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-13.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-13.el7ost.noarch
openstack-puppet-modules-8.1.2-1.el7ost.noarch
instack-undercloud-4.0.0-5.el7ost.noarch



Steps to reproduce:
1. Deploy 8.0 with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1   --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

2. Upgrade the undercloud 
3. Successfully pass the step with "-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml"
4. Attempt to do the keystone wsgi migration step (includes  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-keystone-liberty-mitaka.yaml)
Result:
2016-07-08 19:23:50 [ControllerDeployment]: SIGNAL_COMPLETE Unknown
2016-07-08 19:23:52 [1]: SIGNAL_COMPLETE Unknown
2016-07-08 19:23:54 [1]: SIGNAL_COMPLETE Unknown
Stack overcloud UPDATE_FAILED
Deployment failed:  Heat Stack update failed.

Checked the pcs resource status. Several resources were stopped.
ran "pcs resource cleanup"
All but one resource successfully started. 

[root@overcloud-controller-1 ~]# pcs status|grep -B1 -i stop
 Clone Set: httpd-clone [httpd]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]


Was unable to start this resource:
(98)Address already in use: AH00072: make_sock: could not bind to address 192.168.100.12:5000



turned out "/usr/bin/python2/usr/bin/keystone-all" was already using port 5000.

Comment 2 Alexander Chuzhoy 2016-07-08 20:33:25 UTC
The workaround is to stop the openstack-keystone.service on all controllers and then to start the httpd resource.

Comment 19 errata-xmlrpc 2016-08-11 11:35:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html