Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1693041 - OC deployed with BMaaS + TLS everywhere, : Error contacting Ironic server
Summary: OC deployed with BMaaS + TLS everywhere, : Error contacting Ironic server
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-27 02:17 UTC by Alexander Chuzhoy
Modified: 2019-04-12 02:44 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-12 02:44:12 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Alexander Chuzhoy 2019-03-27 02:17:48 UTC
OC deployed with BMaaS + TLS everywhere,  : Error contacting Ironic server

Environment:
openstack-ironic-staging-drivers-0.9.1-1.el7ost.noarch
python2-ironic-neutron-agent-1.0.0-1.el7ost.noarch
openstack-ironic-api-10.1.7-1.el7ost.noarch
openstack-ironic-inspector-7.2.1-5.el7ost.noarch
puppet-ironic-12.4.0-4.el7ost.noarch
python-ironic-inspector-client-3.1.1-2.el7ost.noarch
instack-undercloud-8.4.6-6.el7ost.noarch
openstack-ironic-conductor-10.1.7-1.el7ost.noarch
python2-ironicclient-2.2.1-1.el7ost.noarch
python-ironic-lib-2.12.1-2.el7ost.noarch
openstack-ironic-common-10.1.7-1.el7ost.noarch


Steps to reproduce:

1. deploy ironic in overcloud with TLS everywhere
2. Attempt to run any command against ironic

Result:
(overcloud) [stack@undercloud-0 ~]$ openstack baremetal node list                                                                                            
Error contacting Ironic server: Unable to establish connection to https://overcloud.redhat.local:13385/v1/nodes: ('Connection aborted.', BadStatusLine("''",)). Attempt 6 of 6
Unable to establish connection to https://overcloud.redhat.local:13385/v1/nodes: ('Connection aborted.', BadStatusLine("''",))



Looking at the relevant haproxy.cfg part:

listen ironic
  bind 10.0.0.101:13385 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 192.168.24.101:6385 transparent ssl crt /etc/pki/tls/certs/haproxy/overcloud-haproxy-ctlplane.pem
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  option httpchk
  option httplog
  server controller-0.ctlplane.redhat.local 192.168.24.15:6385 check fall 5 inter 2000 rise 2 verifyhost controller-0.ctlplane.redhat.local
  server controller-1.ctlplane.redhat.local 192.168.24.7:6385 check fall 5 inter 2000 rise 2 verifyhost controller-1.ctlplane.redhat.local
  server controller-2.ctlplane.redhat.local 192.168.24.8:6385 check fall 5 inter 2000 rise 2 verifyhost controller-2.ctlplane.redhat.local



Made sure I'm able to reach all ports/ips from above.
Both certificates seem valid.

Comment 3 Dmitry Tantsur 2019-03-27 08:53:41 UTC
Are you sure you have https://review.openstack.org/645118 applied? I'm not sure what it does, just asking.

Are you able to cURL https://overcloud.redhat.local:13385/v1/nodes? With python-requests? Could you paste the content of your overcloudrc (without passwords)?

Comment 4 Alexander Chuzhoy 2019-03-27 13:28:40 UTC
I applied https://code.engineering.redhat.com/gerrit/#/c/165870/ , what seems to be downstream for https://review.openstack.org/645118


(overcloud) [stack@undercloud-0 ~]$ cat overcloudrc 
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_NO_CACHE=True
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,overcloud.redhat.local,overcloud.ctlplane.redhat.local
export OS_USER_DOMAIN_NAME=Default
export OS_VOLUME_API_VERSION=3
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=https://overcloud.redhat.local:13000//v3
export NOVA_VERSION=1.1
export OS_IMAGE_API_VERSION=2
export OS_PASSWORD=<pass>
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_PROJECT_NAME=admin
export OS_AUTH_TYPE=password
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"

# Add OS_CLOUDNAME to PS1
if [ -z "${CLOUDPROMPT_ENABLED:-}" ]; then
    export PS1=${PS1:-""}
    export PS1=\${OS_CLOUDNAME:+"(\$OS_CLOUDNAME)"}\ $PS1
    export CLOUDPROMPT_ENABLED=1
fi

(overcloud) [stack@undercloud-0 ~]$ curl https://overcloud.redhat.local:13385/v1/nodes
curl: (52) Empty reply from server
(overcloud) [stack@undercloud-0 ~]$

Comment 5 Alexander Chuzhoy 2019-03-27 13:36:05 UTC
(overcloud) [stack@undercloud-0 ~]$ cat foo.py 
import requests

response = requests.get('https://overcloud.redhat.local:13385/v1/nodes')
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ python foo.py 
Traceback (most recent call last):
  File "foo.py", line 3, in <module>
    response = requests.get('https://overcloud.redhat.local:13385/v1/nodes')
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 488, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Comment 6 Alexander Chuzhoy 2019-03-28 15:41:20 UTC
Using the procedure described here: http://hardysteven.blogspot.com/2016/08/tripleo-deploy-artifacts-and-puppet.html


I re-deployed OC ensuring  https://review.openstack.org/#/c/645118/1/manifests/haproxy.pp is added on OC nodes. 
The issue was resolved by the above operation.

Comment 7 Bob Fournier 2019-04-01 17:58:11 UTC
Sasha - can we close this?

Comment 9 Bob Fournier 2019-04-11 15:53:26 UTC
Can we retest with 13z6?

Comment 10 Alexander Chuzhoy 2019-04-12 02:44:12 UTC
Environment:
openstack-tripleo-heat-templates-8.3.1-5.el7ost.noarch
puppet-tripleo-8.4.1-2.el7ost.noarch


The reported issue doesn't reproduce.


Note You need to log in before you can comment on or make changes to this bug.