Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1689263 - Internal error: Unable to set dialer for kube-service-catalog
Summary: Internal error: Unable to set dialer for kube-service-catalog
Keywords:
Status: CLOSED DUPLICATE of bug 1680342
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Catalog
Version: 3.11.0
Hardware: Unspecified
OS: Linux
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Jay Boyd
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-15 14:24 UTC by szustkowski
Modified: 2019-03-19 11:44 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-19 11:44:23 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description szustkowski 2019-03-15 14:24:38 UTC
Description of problem:

Possibly similar to https://bugzilla.redhat.com/show_bug.cgi?id=1669131. An installation of OpenShift 3.11 fails when 

"Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered"

The error is: 

"Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserve
r as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding

===

Version-Release number of the following components:
openshift-ansible-3.11.92-1.git.0.f2fade7.el7.noarch
ansible-2.6.14-1.el7ae.noarch

ansible 2.6.14
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/tu_ose_ansible/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

===

How reproducible:

Steps to Reproduce:
1. Try to install OpenShift 3.11 from scratch (doesnt matter if ovs-multitenant is enabled or disabled), with enabled service catalog. 
2. Wait for the installer to get to this particular task
3.

Actual results:
TASK [openshift_control_plane : Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered] *************************************************************************************************************************************************
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (30 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (30 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (30 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (29 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (29 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (29 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (28 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (28 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (28 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (27 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (27 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (27 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (26 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (26 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (26 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (25 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (25 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (25 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (24 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (24 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (24 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (23 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (23 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (23 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (22 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (22 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (22 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (21 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (21 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (21 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (20 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (20 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (20 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (19 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (19 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (19 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (18 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (18 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (18 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (17 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (17 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (17 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (16 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (16 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (16 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (15 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (15 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (15 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (14 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (14 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (14 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (13 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (13 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (13 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (12 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (12 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (12 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (11 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (11 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (11 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (10 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (10 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (10 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (9 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (9 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (8 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (9 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (8 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (7 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (8 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (7 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (6 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (7 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (6 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (5 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (6 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (5 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (4 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (5 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (4 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (3 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (4 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (3 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (2 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (3 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (2 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (1 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (2 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (1 retries left).
fatal: [prod-ose3-master2.internal.viada.de]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["oc", "--config=/etc/origin/master/admin.kubeconfig", "get", "--raw", "/apis/servicecatalog.k8s.io/v1beta1"], "delta": "0:00:00.163678", "en
d": "2019-03-15 14:52:41.209982", "msg": "non-zero return code", "rc": 1, "start": "2019-03-15 14:52:41.046304", "stderr": "Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserve
r as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding", "stderr_lines": ["Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding"], "stdout": "", "stdout_lines": []}
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (1 retries left).
fatal: [prod-ose3-master1.internal.viada.de]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["oc", "--config=/etc/origin/master/admin.kubeconfig", "get", "--raw", "/apis/servicecatalog.k8s.io/v1beta1"], "delta": "0:00:00.153208", "end": "2019-03-15 14:52:44.034452", "msg": "non-zero return code", "rc": 1, "start": "2019-03-15 14:52:43.881244", "stderr": "Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding", "stderr_lines": ["Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding"], "stdout": "", "stdout_lines": []}
fatal: [prod-ose3-master3.internal.viada.de]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["oc", "--config=/etc/origin/master/admin.kubeconfig", "get", "--raw", "/apis/servicecatalog.k8s.io/v1beta1"], "delta": "0:00:00.291200", "end": "2019-03-15 14:52:48.847290", "msg": "non-zero return code", "rc": 1, "start": "2019-03-15 14:52:48.556090", "stderr": "Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding", "stderr_lines": ["Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding"], "stdout": "", "stdout_lines": []}

NO MORE HOSTS LEFT ****************************************************************************************************************************************************************************************************************************
 [WARNING]: Could not create retry file '/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry'.         [Errno 13] Permission denied: u'/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry'


PLAY RECAP ************************************************************************************************************************************************************************************************************************************
localhost                  : ok=11   changed=0    unreachable=0    failed=0
prod-ose3-infra1.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-infra2.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-infra3.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-master1.internal.viada.de : ok=299  changed=51   unreachable=0    failed=1   
prod-ose3-master2.internal.viada.de : ok=238  changed=46   unreachable=0    failed=1   
prod-ose3-master3.internal.viada.de : ok=238  changed=46   unreachable=0    failed=1   
prod-ose3-worker1.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-worker2.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-worker3.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
INSTALLER STATUS ******************************************************************************************************************************************************************************************************************************
Initialization              : Complete (0:01:22)
Health Check                : Complete (0:01:15)
Node Bootstrap Preparation  : Complete (0:17:02)
etcd Install                : Complete (0:01:46)
Master Install              : In Progress (0:07:16)
        This phase can be restarted by running: playbooks/openshift-master/config.yml


Failure summary:


  1. Hosts:    prod-ose3-master1.internal.viada.de, prod-ose3-master2.internal.viada.de, prod-ose3-master3.internal.viada.de
     Play:     Configure masters
     Task:     Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered
     Message:  non-zero return code

===

Expected results:
This task should succeed without errors.

Comment 1 Jay Boyd 2019-03-15 14:57:03 UTC
This error:

Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding"

is a pretty solid indication you are hitting the same issue described in https://bugzilla.redhat.com/show_bug.cgi?id=1680342.

There is a bug in core Kubernetes (https://github.com/kubernetes/kubernetes/issues/65848) where if the verbosity level for the Kube API Server is configured >5 it will cause this error and log the error "unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper"

Can you please check the configuration and verify the log level?  And change loglevel < 5 if applicable?

Comment 2 szustkowski 2019-03-19 08:08:59 UTC
Hey Jay,

thanks for your help. This was exactly the issue. I have set the loglevel to 4 (it was 6 when i opened this bugreport), and the installation succeeded. So, it is a duplicate of #1680342. 

Best regards, Simon

Comment 3 Jay Boyd 2019-03-19 11:44:23 UTC
Excellent, thanks Simon.

*** This bug has been marked as a duplicate of bug 1680342 ***


Note You need to log in before you can comment on or make changes to this bug.