Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1693808 - Creating TERMINATED_HTTPS Octavia loadbalancer listener fails with 'Could not retrieve certificate' after z5 update
Summary: Creating TERMINATED_HTTPS Octavia loadbalancer listener fails with 'Could not...
Keywords:
Status: ON_DEV
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z6
: 13.0 (Queens)
Assignee: Carlos Goncalves
QA Contact: Bruna Bonguardo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-28 16:34 UTC by Kellen Gattis
Modified: 2019-04-16 13:33 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
Updates octavia-base to use v3 endpoint for service_auth url (deleted)
2019-03-28 16:34 UTC, Kellen Gattis
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Launchpad 1824777 None None None 2019-04-15 09:28:29 UTC
OpenStack gerrit 652603 None None None 2019-04-15 09:28:29 UTC

Description Kellen Gattis 2019-03-28 16:34:50 UTC
Created attachment 1549132 [details]
Updates octavia-base to use v3 endpoint for service_auth url

Description of problem:
After updating to the latest z5 release of RHOSP 13, trying to create a TERMINATED_HTTPS loadbalancer listener is failing with 'Could not retrieve certificate: ['<HATEOAS>'] (HTTP 400) (Request-ID: req-<uuid>)' where <HATEOAS> is the secret URL and <uuid> is the uuid of the request.


Version-Release number of selected component (if applicable):
registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-66
python-octavia-2.0.3-2.el7ost.noarch
puppet-octavia-12.4.0-8.el7ost.noarch
openstack-octavia-common-2.0.3-2.el7ost.noarch
openstack-octavia-api-2.0.3-2.el7ost.noarch
openstack-tripleo-heat-templates-8.2.0-6.1.el7ost.noarch

How reproducible:
Easy to reproduce.
Following the same series of steps against the scenario when registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-64 is the image used by octavia_api works, but fails with 13.0-66.

Steps to Reproduce:
Standard steps for creating a TERMINATED_HTTPS loadbalancer -

openssl pkcs12 -export -inkey server.key -in server.crt -certfile ca-chain.crt -passout pass: -out server.p12
openstack secret store --name='tls_secret1' -t 'application/octet-stream' -e 'base64' --payload="$(base64 < server.p12)"
openstack acl user add -u <octavia_user_uuid> $(openstack secret list | awk '/ tls_secret1 / {print $2}')
openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=$(openstack secret list | awk '/ tls_secret1 / {print $2}') lb1

Actual results:
Creation fails with: 'Could not retrieve certificate: ['<HATEOAS>'] (HTTP 400) (Request-ID: req-<uuid>)' where <HATEOAS> is the secret URL and <uuid> is the uuid of the request

Expected results:
Loadbalancer listener is created

Additional info:
The new version of the Octavia image looks like it includes some enhancements that are designed to automatically add the octavia user uuid to the barbican secret containing the certificate data
Presumably, this is related to this bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1569129

From what I can tell, the issue with creating a TERMINATED_HTTPS listener against the new code is that the auth_url in the [service_auth] section of octavia.conf is being configured by tripleo to use KeystonePublic as the endpoint and not KeystoneV3Public.
Hand-editing octavia.conf and adding "/v3" to the auth_url, restarting the service, and running through the 'steps to reproduce' results in a successful loadbalancer listener creation.



The value for auth_url comes from the following line in /usr/share/openstack-tripleo-heat-templates/puppet/services/octavia-base.yaml:
             octavia::service_auth::auth_url: {get_param: [EndpointMap, KeystonePublic, uri]}


I've attached a patch that simply updates KeystonePublic to KeystoneV3Public.

Comment 1 Kellen Gattis 2019-03-28 17:31:21 UTC
Additional information:

From the perspective of the Octavia API log file, this is what the error looked like:

2019-03-27 19:34:03.042 1 DEBUG octavia.certificates.manager.barbican [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Setting project ACL for certificate secret... set_acls /usr/lib/python2.7/site-packages/octavia/certificates/manager/barbican.py:148
2019-03-27 19:34:03.747 1 DEBUG barbicanclient.client [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Creating Client object Client /usr/lib/python2.7/site-packages/barbicanclient/client.py:156
2019-03-27 19:34:03.809 1 DEBUG barbicanclient.v1.acls [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Getting ACL for secret href: http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/acl get /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:458
2019-03-27 19:34:04.106 1 DEBUG wsme.api [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222



In an attempt to get a more meaningful error message, I modified /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py and was able to get it to generate this traceback:
2019-03-28 04:44:31.080 1 ERROR root [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] : NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398)
2019-03-28 04:44:31.080 1 ERROR root Traceback (most recent call last):
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py", line 333, in load_acls_data
2019-03-28 04:44:31.080 1 ERROR root     response = self._api.get(self.acl_ref)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 70, in get
2019-03-28 04:44:31.080 1 ERROR root     return super(_HTTPClient, self).get(*args, **kwargs).json()
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
2019-03-28 04:44:31.080 1 ERROR root     return self.request(url, 'GET', **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 62, in request
2019-03-28 04:44:31.080 1 ERROR root     resp = super(_HTTPClient, self).request(*args, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 189, in request
2019-03-28 04:44:31.080 1 ERROR root     return self.session.request(url, method, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 573, in request
2019-03-28 04:44:31.080 1 ERROR root     auth_headers = self.get_auth_headers(auth)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 900, in get_auth_headers
2019-03-28 04:44:31.080 1 ERROR root     return auth.get_headers(self, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
2019-03-28 04:44:31.080 1 ERROR root     token = self.get_token(session)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
2019-03-28 04:44:31.080 1 ERROR root     return self.get_access(session).auth_token
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2019-03-28 04:44:31.080 1 ERROR root     self.auth_ref = self.get_auth_ref(session)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref
2019-03-28 04:44:31.080 1 ERROR root     return self._plugin.get_auth_ref(session, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref
2019-03-28 04:44:31.080 1 ERROR root     authenticated=False, log=False)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 848, in post
2019-03-28 04:44:31.080 1 ERROR root     return self.request(url, 'POST', **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 737, in request
2019-03-28 04:44:31.080 1 ERROR root     raise exceptions.from_response(resp, method, url)
2019-03-28 04:44:31.080 1 ERROR root NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398)
2019-03-28 04:44:31.080 1 ERROR root
2019-03-28 04:44:31.152 1 DEBUG barbicanclient.v1.acls [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] After _api.get load_acls_data /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:336
2019-03-28 04:44:31.162 1 DEBUG wsme.api [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222

Comment 2 Carlos Goncalves 2019-04-15 08:49:31 UTC
Kellen, thank you so much for the detailed report, root cause analysis and proposed patch!

Since the auth URL is used between services, I think KeystoneV3Internal would be even more adequate. Let me know if you disagree. In the mean time I'll go ahead and propose the fix upstream.

Comment 6 Kellen Gattis 2019-04-16 13:33:48 UTC
KeystoneV3Internal makes sense to me.

Thanks!


Note You need to log in before you can comment on or make changes to this bug.