Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1688503 - openshift-apiserver goes Available=False after 4hrs
Summary: openshift-apiserver goes Available=False after 4hrs
Keywords:
Status: CLOSED DUPLICATE of bug 1684547
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 4.1
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.1.0
Assignee: Michal Fojtik
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-13 21:06 UTC by Seth Jennings
Modified: 2019-03-18 14:35 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-18 14:35:53 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1688147 None NEW Service discovery is broken for a several hours cluster 2019-03-18 12:10:52 UTC

Description Seth Jennings 2019-03-13 21:06:10 UTC
$ oc get clusterversion
NAME      VERSION                           AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.alpha-2019-03-13-010143   True        False         5h58m   Cluster version is 4.0.0-0.alpha-2019-03-13-010143

$ oc get clusteroperators openshift-apiserver
NAME                  VERSION                           AVAILABLE   PROGRESSING   FAILING   SINCE
openshift-apiserver   4.0.0-0.alpha-2019-03-13-010143   False       False         False     133m

$ oc get clusteroperators openshift-apiserver -oyaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-03-13T14:59:59Z
  generation: 1
  name: openshift-apiserver
  resourceVersion: "158276"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/openshift-apiserver
  uid: ac303a76-45a0-11e9-8640-0651e81f5f5c
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-03-13T16:50:42Z
    reason: AsExpected
    status: "False"
    type: Failing
  - lastTransitionTime: 2019-03-13T15:02:19Z
    reason: AsExpected
    status: "False"
    type: Progressing
  - lastTransitionTime: 2019-03-13T18:52:06Z
    message: |-
      Available: v1.apps.openshift.io is not ready: 401
      Available: v1.authorization.openshift.io is not ready: 401
      Available: v1.build.openshift.io is not ready: 401
      Available: v1.image.openshift.io is not ready: 401
      Available: v1.oauth.openshift.io is not ready: 401
      Available: v1.project.openshift.io is not ready: 401
      Available: v1.quota.openshift.io is not ready: 401
      Available: v1.route.openshift.io is not ready: 401
      Available: v1.security.openshift.io is not ready: 401
      Available: v1.template.openshift.io is not ready: 401
      Available: v1.user.openshift.io is not ready: 401
    reason: Available
    status: "False"
    type: Available
  - lastTransitionTime: 2019-03-13T14:59:59Z
    reason: NoData
    status: Unknown
    type: Upgradeable
  extension: null
  relatedObjects:
  - group: operator.openshift.io
    name: cluster
    resource: openshiftapiservers
  - group: ""
    name: openshift-config
    resource: namespaces
  - group: ""
    name: openshift-config-managed
    resource: namespaces
  - group: ""
    name: openshift-apiserver-operator
    resource: namespaces
  - group: ""
    name: openshift-apiserver
    resource: namespaces
  - group: apiregistration.k8s.io
    name: v1.apps.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.authorization.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.build.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.image.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.oauth.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.project.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.quota.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.route.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.security.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.template.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.user.openshift.io
    resource: apiservices
  versions:
  - name: operator
    version: 4.0.0-0.alpha-2019-03-13-010143
  - name: openshift-apiserver
    version: 4.0.0-0.alpha-2019-03-13-010143_openshift

$ oc logs -n openshift-apiserver apiserver-7qphx | tail -n20
E0313 21:05:18.574628       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:18.581683       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:18.592547       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.367177       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.399546       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.437282       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.439531       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.488737       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.543609       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.545969       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:19.550761       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.212368       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.219074       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.401941       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.406456       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.490426       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.546941       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.547837       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.551435       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
E0313 21:05:20.552280       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]

Comment 1 Seth Jennings 2019-03-13 21:08:51 UTC
I think this is because the apiserver no longer trusted the CA that signed the cert that openshift-authentication is using.

Comment 2 Xingxing Xia 2019-03-14 07:51:42 UTC
Same as bug 1688147

Comment 3 Radek Vokal 2019-03-18 14:35:53 UTC

*** This bug has been marked as a duplicate of bug 1684547 ***


Note You need to log in before you can comment on or make changes to this bug.