Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1058291 - openstack-keystone service fails to start if ssl is configured
Summary: openstack-keystone service fails to start if ssl is configured
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-keystone
Version: 4.0
Hardware: All
OS: Linux
high
high
Target Milestone: z2
: 4.0
Assignee: Alan Pevec
QA Contact: Udi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-27 12:56 UTC by Lee Yarwood
Modified: 2018-12-05 17:02 UTC (History)
13 users (show)

Fixed In Version: openstack-keystone-2013.2.2-1.el6ost
Doc Type: Bug Fix
Doc Text:
Previously, openstack-keystone initscript was using "keystone discover" to detect when the Identity Service had finished initialization and was ready to serve. However "keystone discover" hangs when the OpenStack Identity Service is configured to use SSL, causing openstack-keystone initscript to hang on startup. This has been fixed by changing openstack-keystone initscript so that it waits for notification from the Identity Service that it is ready to serve. The keystoneclient discover command is no longer used. As a result, openstack-keystone initscript starts successfully when SSL is configured.
Clone Of:
: 1066608 1066627 (view as bug list)
Environment:
Last Closed: 2014-03-04 20:15:36 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 699353 None None None Never
Launchpad 1270154 None None None Never
Red Hat Product Errata RHBA-2014:0213 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform 4 Bug Fix and Enhancement Advisory 2014-03-05 01:11:55 UTC
OpenStack gerrit 67474 None None None Never

Description Lee Yarwood 2014-01-27 12:56:07 UTC
Description of problem:

The openstack-keystone service fails to start if keystone is configured to serve https on 35357.

The following upstream bugs and changes have been opened to correct this :

https://bugs.launchpad.net/python-keystoneclient/+bug/1270154
https://review.openstack.org/67474

Downstream the following patch works around this issue for now :

~~~
--- client.py.orig	2014-01-22 16:08:14.553000854 +0000
+++ client.py	2014-01-23 09:02:32.289998111 +0000
@@ -75,12 +75,15 @@
 
     def _local_keystone_exists(self):
         """Checks if Keystone is available on default local port 35357."""
-        return self._check_keystone_versions("http://localhost:35357")
+	results = self._check_keystone_versions("http://localhost:35357")
+	if results is None:
+		results = self._check_keystone_versions("https://localhost:35357")
+	return results	
 
-    def _check_keystone_versions(self, url):
+    def _check_keystone_versions(self, url, timeout=5):
         """Calls Keystone URL and detects the available API versions."""
         try:
-            client = httpclient.HTTPClient()
+            client = httpclient.HTTPClient(insecure=True, timeout=timeout)
             resp, body = client.request(url, "GET",
                                         headers={'Accept':
                                                  'application/json'})
@@ -140,10 +143,10 @@
         if url:
             return self._check_keystone_extensions(url)
 
-    def _check_keystone_extensions(self, url):
+    def _check_keystone_extensions(self, url, timeout=5):
         """Calls Keystone URL and detects the available extensions."""
         try:
-            client = httpclient.HTTPClient()
+            client = httpclient.HTTPClient(insecure=True,timeout=timeout)
             if not url.endswith("/"):
                 url += '/'
             resp, body = client.request("%sextensions" % url, "GET",
~~~

Version-Release number of selected component (if applicable):
openstack-keystone-2013.2-3.el6ost.noarch                   Wed Jan 22 14:47:22 2014
python-keystone-2013.2-3.el6ost.noarch                      Wed Jan 22 14:47:21 2014
python-keystoneclient-0.4.1-3.el6ost.noarch                 Wed Jan 22 14:46:40 2014

How reproducible:
Always.

Steps to Reproduce:
1. Configure keystone to serve https only over the default admin port of 35357.
2. Attempt to start the openstack-keystone service.


Actual results:
Service hangs attempting to check if a local keystone instance is already running.

Expected results:
Service is able to pass this check and start.

Additional info:

Comment 2 Jamie Lennox 2014-01-28 02:05:45 UTC
I just approved the linked review which will i think provide only a section of this bug. 

Review: https://review.openstack.org/#/c/67474/ will make it so that the discover command works like every other keystone command and will now respond to the --insecure and --timeout flags. 

As the cause of this bug is in keystoneclient it will not follow the openstack release cycle - however i am not aware of when the next release is expected. 

These flags will need to be added to the whatever is calling the keystoneclient script. They can be added before the client is released, and they will simply be ignored until. 

From looking at https://github.com/redhat-openstack/openstack-keystone/blob/master/openstack-keystone.service I can't see where the initial call is happening though. Am i looking at too new a version? Do you know where the discover call is coming from?

Comment 3 Lee Yarwood 2014-01-28 09:48:06 UTC
(In reply to Jamie Lennox from comment #2)
> I just approved the linked review which will i think provide only a section
> of this bug. 
> 
> Review: https://review.openstack.org/#/c/67474/ will make it so that the
> discover command works like every other keystone command and will now
> respond to the --insecure and --timeout flags. 
>
> As the cause of this bug is in keystoneclient it will not follow the
> openstack release cycle - however i am not aware of when the next release is
> expected. 

While the issue is with the client this does make it appear that the overall openstack-keystone service has hung at startup, would that change how we address this issue?

> These flags will need to be added to the whatever is calling the
> keystoneclient script. They can be added before the client is released, and
> they will simply be ignored until.
>
> From looking at
> https://github.com/redhat-openstack/openstack-keystone/blob/master/openstack-
> keystone.service I can't see where the initial call is happening though. Am
> i looking at too new a version? Do you know where the discover call is
> coming from?

Ah yes, my apologies for not clearly stating that this was a RHOS 4 issue on RHEL 6.5 hosts. Thus the above systemd service file doesn't apply here. The SysVinit file calls and then waits for 'keystone discover' to return causing the service to appear to hang here.

Comment 5 Jamie Lennox 2014-01-28 18:45:02 UTC
It won't change the default behaviour initially, but it will mean that we can add --insecure and --timeout=5 to the discover call in the startup script (we can add them now they'll just be ignored).

--insecure is to not verify a certificate
--timeout is how long before we give up on making a request. 

passing those should have the same effect as the changed parameters in the diff from comment 1.

Comment 7 Nathan Kinder 2014-01-29 00:48:05 UTC
Targeting for A2.  We should cherry-pick the upstream keystoneclient patch.  A patch for the init script needs to be proposed.

Comment 8 Alan Pevec 2014-01-30 15:01:54 UTC
(In reply to Nathan Kinder from comment #7)
> Targeting for A2.  We should cherry-pick the upstream keystoneclient patch. 

Changing component for this part.

> A patch for the init script needs to be proposed.

This part is obsoleted by bug 1036515 comment 21 change in openstack-keystone, also targeting A2.

Comment 11 Jamie Lennox 2014-02-11 22:29:43 UTC
Keystone client released with discover fix, i think it was 0.5.1 (might have been 0.5.0). Need the init script timeout fix still.

Comment 12 Alan Pevec 2014-02-12 15:17:41 UTC
(In reply to Jamie Lennox from comment #11)
> Keystone client released with discover fix, i think it was 0.5.1 (might have
> been 0.5.0). Need the init script timeout fix still.

We can't rebase to >= 0.4.2 as it breaks puppet-keystone module which still uses deprecated --endpoint option which was removed in 0.4.2.

Comment 18 Jamie Lennox 2014-02-17 00:54:19 UTC
So puppet-keystone should be updated to use --os-endpoint instead of --endpoint which has been supported for a while (though i can't say exactly when --os-endpoint was added).

I'm not sure if this has been filed elsewhere so just putting needinfo to rob as i think that puppet-keystone is his department.

Comment 19 Rob Crittenden 2014-02-17 21:39:34 UTC
The switch to --os-endpoint is under review now at https://review.openstack.org/#/c/73970/

IMHO --insecure should be avoided.

Comment 20 Alan Pevec 2014-02-18 19:32:59 UTC
(In reply to Rob Crittenden from comment #19)
> The switch to --os-endpoint is under review now at
> https://review.openstack.org/#/c/73970/

Filed bug 1066627 for openstack-puppet-modules

Comment 22 Udi 2014-02-19 11:40:57 UTC
The patch provided in this bug is not present in the Havana A2 release. I still checked if the service starts when configured with SSL - and it started successfully so somehow it is fixed another way.

Hovever, I tried a simple "keystone user-list" command and it hanged indefinitely. When I pressed Ctrl+C I got this exception:

  File "/usr/bin/keystone", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/keystoneclient/shell.py", line 490, in main
    OpenStackIdentityShell().main(sys.argv[1:])
  File "/usr/lib/python2.6/site-packages/keystoneclient/shell.py", line 433, in main
    timeout=args.timeout)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 139, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 468, in authenticate
    resp, body = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 162, in get_raw_token_from_identity_service
    token=token)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 197, in _base_authN
    resp, body = self.request(url, 'POST', body=params, headers=headers)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 610, in request
    **request_kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 112, in request
    **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 174, in send
    timeout=timeout
  File "/usr/lib/python2.6/site-packages/urllib3/connectionpool.py", line 427, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.6/site-packages/urllib3/connectionpool.py", line 289, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status
    line = self.fp.readline()
  File "/usr/lib64/python2.6/socket.py", line 433, in readline
    data = recv(1)
KeyboardInterrupt


I tried to change OS_AUTH_URL in keystonerc_admin to look for https://... and it didn't work at all - it complained that it can't establish a connection.

Should I fail this bug? I need an explanation on how this bug was fixed if it wasn't fixed using our patch, and how to succeed with the client when keystone runs with ssl enabled.

By the way, to configure keystone with ssl, I uncommented the following lines in keystone.conf:
[ssl]
enable = True
certfile = /etc/keystone/pki/certs/ssl_cert.pem
keyfile = /etc/keystone/pki/private/ssl_key.pem
ca_certs = /etc/keystone/pki/certs/cacert.pem
ca_key = /etc/keystone/pki/private/cakey.pem

And then I ran the command:
keystone-manage ssl_setup --keystone-user keystone --keystone-group keystone

.. and restarted the service. Is that the right procedure?

Comment 23 Nathan Kinder 2014-02-20 01:31:41 UTC
(In reply to Udi from comment #22)
> The patch provided in this bug is not present in the Havana A2 release. I
> still checked if the service starts when configured with SSL - and it
> started successfully so somehow it is fixed another way.
> 
...
> 
> Should I fail this bug? I need an explanation on how this bug was fixed if
> it wasn't fixed using our patch, and how to succeed with the client when
> keystone runs with ssl enabled.

I believe that Alan fixed it in the Keystone init script as mentioned here:

    https://bugzilla.redhat.com/show_bug.cgi?id=1036515#c26

Comment 24 Nathan Kinder 2014-02-20 01:46:44 UTC
(In reply to Udi from comment #22)
> The patch provided in this bug is not present in the Havana A2 release. I
> still checked if the service starts when configured with SSL - and it
> started successfully so somehow it is fixed another way.
> 
> Hovever, I tried a simple "keystone user-list" command and it hanged
> indefinitely.

This hang sounds normal for a non-SSL client that tries to connect to a SSL enabled server.  That makes me think that you have the server properly configured for SSL.

> 
> I tried to change OS_AUTH_URL in keystonerc_admin to look for https://...
> and it didn't work at all - it complained that it can't establish a
> connection.

You should be specifying "https://".  Would you supply the exact OS_AUTH_URL that you are using along with the exact error that it reports?

It's quite possible that the certificate is not being validated properly due to a hostname mismatch between the CN in the certificate subject DN and the hostname that you are using in OS_AUTH_URL.  The "keystone" client has a "--insecure" option that skips certificate trust checking, but I'm not sure it that would skip a hostname mismatch.  It would be a better test to make sure we have a valid SSL setup with correct hostnames.

Comment 25 Udi 2014-02-20 09:40:51 UTC
Alan's fix does not concern keystone, but nova. So, I still don't see exactly how the bug was fixed but anyways the service starts.

I use the following OS_AUTH_URL:
export OS_AUTH_URL=https://127.0.0.1:35357/v2.0/

I get this error:
keystone user-list
Authorization Failed: Unable to establish connection to https://127.0.0.1:35357/v2.0/tokens

I also tried replacing 127.0.0.1 with the real IP (actually that was the default) and it returns the same error. With the --insecure option I get:
Unable to establish connection to http://10.8.0.50:35357/v2.0/users

Should the bug be marked as verified even though I can't authenticate with the client, and can't authenticate with the API (curl) either?

Comment 28 Alan Pevec 2014-02-20 09:46:10 UTC
(In reply to Udi from comment #25)
> I also tried replacing 127.0.0.1 with the real IP (actually that was the
> default) and it returns the same error. With the --insecure option I get:
> Unable to establish connection to http://10.8.0.50:35357/v2.0/users

You should change identity endpoint in service catalog to use https.

Comment 29 Udi 2014-02-24 15:49:06 UTC
Verified in: python-keystoneclient-0.4.1-4.el6ost.noarch

Service started with SSL enabled. Changed the endpoint urls to https://... and was able to list the catalog with 'keystone --insecure catalog'.

apevec, can you explain why the --insecure flag is needed? Thanks.

Comment 30 Alan Pevec 2014-02-24 16:31:30 UTC
> apevec, can you explain why the --insecure flag is needed?

  --insecure            Explicitly allow keystoneclient to perform "insecure"
                        TLS (https) requests. The server's certificate will
                        not be verified against any certificate authorities.
                        This option should be used with caution.

Steps to reproduce were:
1) uncomment the lines from keystone.conf under the [ssl] section
2) run keystone-manage ssl_setup --keystone-user keystone --keystone-group keystone

^^^ here is self-signed certificate created, hence --insecure

3) update endpoint.url (http: -> https:) in keystone database
4) restart the openstack-keystone service
5) change OS_AUTH_URL to https: in keystonerc_admin and re-source it

Comment 32 errata-xmlrpc 2014-03-04 20:15:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0213.html


Note You need to log in before you can comment on or make changes to this bug.