Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1518306 - Errors performing a overcloud OSP9 minor update from a OSP10 undercloud
Summary: Errors performing a overcloud OSP9 minor update from a OSP10 undercloud
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Lukas Bezdicka
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-28 15:28 UTC by Eduard Barrera
Modified: 2018-09-17 13:09 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-11 15:00:42 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1626422 None None None 2017-11-28 15:40:31 UTC

Description Eduard Barrera 2017-11-28 15:28:32 UTC
Description of problem:


As referenced in the support matrix for RH OSP10, an OSP10 Director should be able to work with an OSP9 Overcloud.

Each time this OSP9 minor update process is started, it very quickly fails with;

openstack overcloud update stack overcloud -i --templates /usr/share/openstack-tripleo-heat-templates/mitaka -e /usr/share/openstack-tripleo-heat-templates/mitaka/overcloud-resource-registry-puppet.yaml -e /home/stack/tripleo-deployment/osp9-production.yaml -e /usr/share/openstack-tripleo-heat-templates/mitaka/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/mitaka/environments/network-management-pool.yaml -e enable-tls-osp9.yaml -e keystone-keys.yaml --log-file logs/osp9-minor-update-2017-11-24.log
starting package update on stack overcloud
ERROR: Failed to validate: : resources.ControllerServiceChain: : Failed to validate nested template: Property error: resources[10].properties: Property KeystoneCredential0 not assigned

The compatibility templates are being used, please note --templates /usr/share/openstack-tripleo-heat-templates/mitaka  on the deployment command


As far as we can tell, this parameter KeystoneCredential0 is introduced in the OSP10 templates but is not present at all in OSP9, and so shouldn't be required.

I can see that 2 Keystone credential keys (KeystoneCredential0 and KeystoneCredential1) have been written to /etc/keystone/credential-keys;

[root@osp-director-prod services]# ls -la /etc/keystone/credential-keys/
total 12
drwxr-xr-x. 2 keystone keystone   22 Nov 24 14:03 .
drwxr-x---. 4 root     keystone 4096 Nov 23 15:28 ..
-rw-------. 1 keystone keystone   44 Nov 23 15:28 0
-rw-------. 1 keystone keystone   44 Nov 23 15:28 1

We have tried including an additional parameter file (keystone-keys.yaml) to the minor update process, specifying these keys, but the process fails at the same point;

[stack@osp-director-prod tripleo-deployment]$ cat keystone-keys.yaml
parameter_defaults:
  KeystoneCredential0: 'REMOVED'
  KeystoneCredential1: 'REMOVED'



Version-Release number of selected component (if applicable):

RH OSP9 Overcloud 3 controller nodes, 3 Ceph storage nodes, 68 compute nodes
RH OSP10 Undercloud Director


How reproducible:
Always

Steps to Reproduce:
1. with the described versions of undercloud/overcloud do a minor update to the undercloud
openstack overcloud update stack overcloud -i --templates /usr/share/openstack-tripleo-heat-templates/mitaka -e /usr/share/openstack-tripleo-heat-templates/mitaka/overcloud-resource-registry-puppet.yaml -e /home/stack/tripleo-deployment/osp9-production.yaml -e /usr/share/openstack-tripleo-heat-templates/mitaka/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/mitaka/environments/network-management-pool.yaml -e enable-tls-osp9.yaml -e keystone-keys.yaml --log-file logs/osp9-minor-update-2017-11-24.log

2.
3.

Actual results:
starting package update on stack overcloud
ERROR: Failed to validate: : resources.ControllerServiceChain: : Failed to validate nested template: Property error: resources[10].properties: Property KeystoneCredential0 not assigned

Expected results:
overcloud updated

Comment 1 Eduard Barrera 2017-11-28 15:40:11 UTC
The symptoms are really similar to:

https://bugs.launchpad.net/tripleo/+bug/1626422

In my "lab" I have the same package version as customer and the patch is there, at least on 
https://review.openstack.org/#/c/374892/6/tripleoclient/tests/v1/overcloud_deploy/test_overcloud_deploy.py

python-tripleoclient-5.4.3-1.el7ost.noarch                  Thu Nov 23 14:46:36 2017

Perhaps I overlooked something.

I traced the error to this mistral action:


$ mistral --debug run-action tripleo.parameters.generate_passwords
DEBUG (v2) Making authentication request to http://10.2.100.1:5000/v2.0/tokens
DEBUG (extension) found extension EntryPoint.parse('yaml = clifftablib.formatters:YamlFormatter')
DEBUG (extension) found extension EntryPoint.parse('json = clifftablib.formatters:JsonFormatter')
DEBUG (extension) found extension EntryPoint.parse('html = clifftablib.formatters:HtmlFormatter')
DEBUG (extension) found extension EntryPoint.parse('table = cliff.formatters.table:TableFormatter')
DEBUG (extension) found extension EntryPoint.parse('json = cliff.formatters.json_format:JSONFormatter')
DEBUG (extension) found extension EntryPoint.parse('shell = cliff.formatters.shell:ShellFormatter')
DEBUG (extension) found extension EntryPoint.parse('value = cliff.formatters.value:ValueFormatter')
DEBUG (extension) found extension EntryPoint.parse('yaml = cliff.formatters.yaml_format:YAMLFormatter')
DEBUG (command) run(Namespace(columns=[], formatter='table', input=None, max_width=0, name='tripleo.parameters.generate_passwords', noindent=False, prefix='', run_sync=False, save_result=False, target=None, variables=[]))
DEBUG (httpclient) HTTP POST http://10.2.100.1:8989/v2/action_executions 201
{"result": "Failed to run action [action_ex_id=None, action_cls='<class 'mistral.actions.action_factory.GeneratePasswordsAction'>', attributes='{}', params='{u'container': u'overcloud'}']\n ERROR: You are not authorized to use environment."}


On mistral logs:

2017-11-27 22:52:08.385 14213 DEBUG requests.packages.urllib3.connectionpool [-] "GET /v1/dfa0a61a731b4b1aad8693d8d65d8a59/stacks/overcloud/97988921-d44f-4757-8508-5fde79e5ad5c/environment HTTP/1.1" 403 197 _make_request /usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:401
2017-11-27 22:52:08.386 14213 DEBUG heatclient.common.http [-]
HTTP/1.1 403 Forbidden
Content-Type: application/json; charset=UTF-8
Content-Length: 197
X-Openstack-Request-Id: req-ddb88e7f-31db-4ba9-a651-66d9f2c68495
Date: Mon, 27 Nov 2017 22:52:08 GMT
Connection: keep-alive

{"explanation": "Access was denied to this resource.", "code": 403, "error": {"message": "You are not authorized to use environment.", "traceback": null, "type": "Forbidden"}, "title": "Forbidden"}
 log_http_response /usr/lib/python2.7/site-packages/heatclient/common/http.py:155
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor [-] Failed to run action [action_ex_id=None, action_cls='<class 'mistral.actions.action_factory.GeneratePasswordsAction'>', attributes='{}', params='{u'container': u'overcloud'}']
 ERROR: You are not authorized to use environment.
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor Traceback (most recent call last):
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor   File "/usr/lib/python2.7/site-packages/mistral/engine/default_executor.py", line 90, in run_action
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor     result = action.run()
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor   File "/usr/lib/python2.7/site-packages/tripleo_common/actions/parameters.py", line 160, in run
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor     stack_id=self.container)
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 293, in environment
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor     resp = self.client.get('/stacks/%s/environment' % stack_id)
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 287, in get
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor     return self.client_request("GET", url, **kwargs)
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 280, in client_request
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor     resp, body = self.json_request(method, url, **kwargs)
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 269, in json_request
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor     resp = self._http_request(url, method, **kwargs)
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 241, in _http_request
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor     resp = self._http_request(location, method, **kwargs)
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 232, in _http_request
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor     raise exc.from_response(resp)
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor HTTPForbidden: ERROR: You are not authorized to use environment.
2017-11-27 22:52:08.387 14213 ERROR mistral.engine.default_executor

and on heat logs:

2017-11-27 22:52:08.200 8823 INFO eventlet.wsgi.server [req-8061ff9d-6c74-43b2-837f-165b476ecfa8 f6a32941ce804aea9ead8129151661f9 dfa0a61a731b4b1aad8693d8d65d8a59 - default default] 10.2.100.1 - - [27/Nov/2017 22:52:08] "GET /v1/dfa0a61a731b4b1aad8693d8d65d8a59/stacks/overcloud/environment HTTP/1.1" 302 744 0.170836
2017-11-27 22:52:08.385 8824 INFO eventlet.wsgi.server [req-ddb88e7f-31db-4ba9-a651-66d9f2c68495 f6a32941ce804aea9ead8129151661f9 dfa0a61a731b4b1aad8693d8d65d8a59 - default default] 10.2.100.1 - - [27/Nov/2017 22:52:08] "GET /v1/dfa0a61a731b4b1aad8693d8d65d8a59/stacks/overcloud/97988921-d44f-4757-8508-5fde79e5ad5c/environment HTTP/1.1" 403 418 0.184409
2017-11-27 22:52:27.304 8822 INFO eventlet.wsgi.server [req-cc26c072-63a5-4c8d-b3f2-7422c05c3922 f6a32941ce804aea9ead8129151661f9 dfa0a61a731b4b1aad8693d8d65d8a59 - default default] 10.2.100.1 - - [27/Nov/2017 22:52:27] "GET /v1/dfa0a61a731b4b1aad8693d8d65d8a59/stacks/overcloud/environment HTTP/1.1" 302 744 0.170399
2017-11-27 22:52:27.449 8824 INFO eventlet.wsgi.server [req-b8da654e-00f4-494f-a6cd-dcdf380b79a0 f6a32941ce804aea9ead8129151661f9 dfa0a61a731b4b1aad8693d8d65d8a59 - default default] 10.2.100.1 - - [27/Nov/2017 22:52:27] "GET /v1/dfa0a61a731b4b1aad8693d8d65d8a59/stacks/overcloud/97988921-d44f-4757-8508-5fde79e5ad5c/environment HTTP/1.1" 403 418 0.144928
/dfa0a61a731b4b1aad8693d8d65d8a59\/stacks\/overcloud\/97988921-d44f-4757-8508-5fde79e5ad5c                                                                                                       12970,230 



I would need some help to trace why the request gets forbidden

Comment 8 Lukas Bezdicka 2017-12-11 15:00:42 UTC
Closing as works for me as this is manual editing of policy.json https://access.redhat.com/solutions/3074511


Note You need to log in before you can comment on or make changes to this bug.