Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1694567 - failed to wait for bootstrap-complete event: openshift-kube-scheduler DoesNotExist
Summary: failed to wait for bootstrap-complete event: openshift-kube-scheduler DoesNot...
Keywords:
Status: ASSIGNED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 4.1
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.1.0
Assignee: Michal Fojtik
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On: 1694223
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-01 06:06 UTC by W. Trevor King
Modified: 2019-04-05 08:28 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description W. Trevor King 2019-04-01 06:06:09 UTC
Description of problem:

An OKD release-promotion gate [1] failed with:

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/e2e-aws/installer/.openshift_install.log | grep -B12 'failed to wait for bootstrap-complete event'
time="2019-03-30T00:54:10Z" level=info msg="API v1.12.4+be1b971 up"
time="2019-03-30T00:54:10Z" level=info msg="Waiting up to 30m0s for the bootstrap-complete event..."
time="2019-03-30T00:54:10Z" level=debug msg="added kube-controller-manager.159096694328a357: ip-10-0-4-79_4def4119-5286-11e9-813b-12611bb1f0b0 became leader"
time="2019-03-30T00:54:11Z" level=debug msg="added kube-scheduler.1590966970efde74: ip-10-0-4-79_4e01f30f-5286-11e9-aebb-12611bb1f0b0 became leader"
time="2019-03-30T01:00:59Z" level=debug msg="added openshift-master-controllers.159096c89df72b29: controller-manager-nx2ks became leader"
time="2019-03-30T01:07:02Z" level=debug msg="added openshift-master-controllers.1590971d0f94f592: controller-manager-nx2ks became leader"
time="2019-03-30T01:14:21Z" level=debug msg="modified kube-controller-manager.159096694328a357: ip-10-0-4-79_4def4119-5286-11e9-813b-12611bb1f0b0 became leader"
time="2019-03-30T01:14:21Z" level=debug msg="modified kube-scheduler.1590966970efde74: ip-10-0-4-79_4e01f30f-5286-11e9-aebb-12611bb1f0b0 became leader"
time="2019-03-30T01:14:21Z" level=debug msg="modified openshift-master-controllers.159096c89df72b29: controller-manager-nx2ks became leader"
time="2019-03-30T01:14:21Z" level=debug msg="modified openshift-master-controllers.1590971d0f94f592: controller-manager-nx2ks became leader"
time="2019-03-30T01:14:38Z" level=debug msg="added kube-controller-manager.15909787548119af: ip-10-0-134-157_c30952d9-5286-11e9-a429-128b6f8916da became leader"
time="2019-03-30T01:14:39Z" level=debug msg="added kube-scheduler.1590978790f22ca9: ip-10-0-4-79_26479e02-5289-11e9-89df-12611bb1f0b0 became leader"
time="2019-03-30T01:24:10Z" level=fatal msg="failed to wait for bootstrap-complete event: timed out waiting for the condition"

Digging into the bootkube.service logs, there were two rounds of cluster-bootstrap:

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/e2e-aws/bootstrap/bootkube.service | grep 'Starting cluster-bootstrap'
Mar 30 00:53:47 ip-10-0-4-79 bootkube.sh[1343]: Starting cluster-bootstrap...
Mar 30 01:14:10 ip-10-0-4-79 bootkube.sh[8920]: Starting cluster-bootstrap...

The first round pushed a bunch of manifests, and then died waiting for openshift-kube-scheduler:

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/e2e-aws/bootstrap/bootkube.service | grep -B5 'timed out waiting for the condition'
Mar 30 00:57:35 ip-10-0-4-79 bootkube.sh[1343]:         Pod Status:openshift-cluster-version/cluster-version-operator        Pending
Mar 30 01:00:35 ip-10-0-4-79 bootkube.sh[1343]:         Pod Status:          kube-apiserver        Ready
Mar 30 01:00:35 ip-10-0-4-79 bootkube.sh[1343]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        DoesNotExist
Mar 30 01:00:35 ip-10-0-4-79 bootkube.sh[1343]:         Pod Status: kube-controller-manager        Ready
Mar 30 01:00:35 ip-10-0-4-79 bootkube.sh[1343]:         Pod Status:openshift-cluster-version/cluster-version-operator        Ready
Mar 30 01:13:50 ip-10-0-4-79 bootkube.sh[1343]: Error: error while checking pod status: timed out waiting for the condition
Mar 30 01:13:50 ip-10-0-4-79 bootkube.sh[1343]: Tearing down temporary bootstrap control plane...
Mar 30 01:13:50 ip-10-0-4-79 bootkube.sh[1343]: Error: error while checking pod status: timed out waiting for the condition

The next round died with "connection refused":

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/e2e-aws/bootstrap/bootkube.service | grep -A2 'failed to create some manifests'
Mar 30 00:54:16 ip-10-0-4-79 bootkube.sh[1343]: [#64] failed to create some manifests:
Mar 30 00:54:16 ip-10-0-4-79 bootkube.sh[1343]: "0000_00_cluster-version-operator_01_clusteroperator.crd.yaml": unable to get REST mapping: no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
Mar 30 00:54:16 ip-10-0-4-79 bootkube.sh[1343]: "0000_00_cluster-version-operator_01_clusterversion.crd.yaml": unable to get REST mapping: no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
--
Mar 30 00:54:21 ip-10-0-4-79 bootkube.sh[1343]: [#65] failed to create some manifests:
Mar 30 00:54:21 ip-10-0-4-79 bootkube.sh[1343]: "cluster-dns-02-config.yml": unable to get REST mapping: no matches for kind "DNS" in version "config.openshift.io/v1"
Mar 30 00:54:21 ip-10-0-4-79 bootkube.sh[1343]: "cluster-infrastructure-02-config.yml": unable to get REST mapping: no matches for kind "Infrastructure" in version "config.openshift.io/v1"
--
Mar 30 01:14:37 ip-10-0-4-79 bootkube.sh[8920]: [#1] failed to create some manifests:
Mar 30 01:14:37 ip-10-0-4-79 bootkube.sh[8920]: "0000_00_cluster-version-operator_00_namespace.yaml": failed to create: Post https://localhost:6443/api/v1/namespaces: dial tcp [::1]:6443: connect: connection refused
Mar 30 01:14:37 ip-10-0-4-79 bootkube.sh[8920]: "0000_00_cluster-version-operator_01_clusteroperator.crd.yaml": failed to create: Post https://localhost:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions: dial tcp [::1]:6443: connect: connection refused

The Kubernetes scheduler was eventually created at 1:19:

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/e2e-aws/pods.json | jq -r '.items[] | select(.metadata.namespace == "openshift-kube-scheduler").status | {startTime, containerStatuses}'
{
  "startTime": "2019-03-30T01:19:53Z",
  "containerStatuses": [
    {
      "image": "registry.svc.ci.openshift.org/ocp/4.0-2019-03-30-003455@sha256:2b119942fe423d50fafeee2968178c050728af3f9fd973de24585a293f15c54d",
      "imageID": "",
      "lastState": {},
      "name": "installer",
      "ready": false,
      "restartCount": 0,
      "state": {
        "waiting": {
          "reason": "ContainerCreating"
        }
      }
    }
  ]
}

but apparently that was too late to save the install.

Version-Release number of selected component (if applicable):

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/release-images-latest/release-images-latest | jq -r .metadata.name
4.0.0-0.ci-2019-03-30-003455
$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/release-images-latest/release-images-latest | jq -r '.spec.tags[] | select(.name == "cluster-kube-scheduler-operator").annotations'
{
  "io.openshift.build.commit.id": "c0eac64757a57e3984cb80cd35add31bc5ff88e7",
  "io.openshift.build.commit.ref": "master",
  "io.openshift.build.source-location": "https://github.com/openshift/cluster-kube-scheduler-operator"
}

How reproducible:

Rare.  Besides the instance I dug into above, this seems to have also happened in [2], with no other instances in the past 48 hours.

[1]: https://prow.k8s.io/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337
[2]: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_cluster-kube-apiserver-operator/360/pull-ci-openshift-cluster-kube-apiserver-operator-master-e2e-aws-operator/199

Comment 1 W. Trevor King 2019-04-01 06:29:24 UTC
There's an outside chance that this is bug 1669532, which was closed INSUFFICIENT_DATA.

Looks like the OpenShift API server was also delayed:

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/e2e-aws/pods.json | jq -r '.items | sort_by(.status.startTime)[] | .status.startTime + " " + .status.phase + " " + (.status.containerStatuses[0].restartCount | tostring) + " " + .metadata.namespace + "/" + .metadata.name' | grep -v ' \(Running\|Succeeded\) 0 '
2019-03-30T00:54:44Z Running 1 openshift-kube-apiserver-operator/kube-apiserver-operator-869cb7fdff-zxdk8
2019-03-30T00:54:45Z Running 1 openshift-apiserver-operator/openshift-apiserver-operator-65f9f5497d-2dn74
2019-03-30T00:54:45Z Running 1 openshift-controller-manager-operator/openshift-controller-manager-operator-7c7b8d745d-49lhc
2019-03-30T00:54:45Z Running 1 openshift-kube-controller-manager-operator/kube-controller-manager-operator-668c78485-sd27b
2019-03-30T00:54:45Z Running 1 openshift-kube-scheduler-operator/openshift-kube-scheduler-operator-579c48744-krlxb
2019-03-30T00:54:52Z Pending 0 openshift-multus/multus-lk9g4
2019-03-30T00:54:57Z Pending 0 openshift-sdn/sdn-controller-pwbqr
2019-03-30T00:54:58Z Pending 0 openshift-sdn/ovs-5w888
2019-03-30T00:54:59Z Pending 0 openshift-sdn/sdn-cxkvz
2019-03-30T00:54:59Z Running 1 openshift-sdn/sdn-wwprm
2019-03-30T00:55:52Z Pending 0 openshift-dns/dns-default-9bzpg
2019-03-30T01:00:52Z Running 1 openshift-controller-manager/controller-manager-nx2ks
2019-03-30T01:14:57Z Running 3 openshift-console/console-7b7c689565-nrv2s
2019-03-30T01:14:57Z Running 3 openshift-console/console-7b7c689565-zz5x2
2019-03-30T01:14:59Z Pending 0 openshift-apiserver/apiserver-xf7pn
2019-03-30T01:14:59Z Pending 0 openshift-controller-manager/controller-manager-8tc4w
2019-03-30T01:19:47Z Pending 0 openshift-authentication-operator/openshift-authentication-operator-9549c876-g7qqs
2019-03-30T01:19:47Z Pending 0 openshift-cluster-node-tuning-operator/cluster-node-tuning-operator-69d8bb4f7f-lgldg
2019-03-30T01:19:47Z Pending 0 openshift-ingress-operator/ingress-operator-5fbc69ff56-8wzsn
2019-03-30T01:19:47Z Pending 0 openshift-marketplace/marketplace-operator-7759c7d45-8vh6f
2019-03-30T01:19:47Z Pending 0 openshift-service-catalog-apiserver-operator/openshift-service-catalog-apiserver-operator-798766f976-jpv9z
2019-03-30T01:19:47Z Pending 0 openshift-service-catalog-controller-manager-operator/openshift-service-catalog-controller-manager-operator-757bz8xvz
2019-03-30T01:19:53Z Pending 0 openshift-kube-controller-manager/installer-3-ip-10-0-163-88.ec2.internal
2019-03-30T01:19:53Z Pending 0 openshift-kube-scheduler/installer-1-ip-10-0-163-88.ec2.internal
2019-03-30T01:19:56Z Pending 0 openshift-kube-apiserver/installer-4-ip-10-0-163-88.ec2.internal
2019-03-30T01:19:58Z Pending 0 openshift-kube-apiserver/revision-pruner-1-ip-10-0-163-88.ec2.internal
2019-03-30T01:24:12Z Pending 0 openshift-operator-lifecycle-manager/packageserver-85c97d676c-tbm4f

Excepts from the OpenShift-API-server operator's logs:

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/e2e-aws/pods/openshift-apiserver-operator_openshift-apiserver-operator-65f9f5497d-2dn74_operator.log.gz | gunzip
...
I0330 01:03:07.877075       1 leaderelection.go:214] successfully acquired lease openshift-apiserver-operator/openshift-apiserver-operator-lock
...
I0330 01:03:07.894741       1 workload_controller.go:158] Starting OpenShiftAPIServerOperator
...
W0330 01:03:07.995444       1 observe_images.go:41] image.config.openshift.io/cluster: not found
...lots of image.config.openshift.io/cluster: not found...
W0330 01:08:12.788518       1 observe_images.go:41] image.config.openshift.io/cluster: not found
I0330 01:08:12.788897       1 status_controller.go:156] clusteroperator/openshift-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2019-03-30T01:03:07Z","reason":"AsExpected","status":"False","type
":"Failing"},{"lastTransitionTime":"2019-03-30T01:03:07Z","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2019-03-30T01:03:07Z","message":"Available: v1.apps.openshift.io is n
ot ready: 503\nAvailable: v1.authorization.openshift.io is not ready: 503\nAvailable: v1.build.openshift.io is not ready: 503\nAvailable: v1.image.openshift.io is not ready: 503\nAvailable: v1.oauth.openshift.io
 is not ready: 503\nAvailable: v1.project.openshift.io is not ready: 503\nAvailable: v1.quota.openshift.io is not ready: 503\nAvailable: v1.route.openshift.io is not ready: 503\nAvailable: v1.security.openshift.
io is not ready: 503\nAvailable: v1.template.openshift.io is not ready: 503\nAvailable: v1.user.openshift.io is not ready: 503","reason":"AvailableMultiple","status":"False","type":"Available"},{"lastTransitionT
ime":"2019-03-30T01:03:07Z","reason":"AsExpected","status":"True","type":"Upgradeable"}]}}
I0330 01:08:12.795250       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-apiserver-operator", Name:"openshift-apiserver-operator", UID:"68d46397-5286-11e9-921d-12611bb1f0b0", 
APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for operator openshift-apiserver changed: Available message changed from "Available: no openshift-a
piserver daemon pods available on any node." to "Available: v1.apps.openshift.io is not ready: 503\nAvailable: v1.authorization.openshift.io is not ready: 503\nAvailable: v1.build.openshift.io is not ready: 503\
nAvailable: v1.image.openshift.io is not ready: 503\nAvailable: v1.oauth.openshift.io is not ready: 503\nAvailable: v1.project.openshift.io is not ready: 503\nAvailable: v1.quota.openshift.io is not ready: 503\n
Available: v1.route.openshift.io is not ready: 503\nAvailable: v1.security.openshift.io is not ready: 503\nAvailable: v1.template.openshift.io is not ready: 503\nAvailable: v1.user.openshift.io is not ready: 503
"
W0330 01:08:14.745168       1 observe_images.go:82] image.config.openshift.io/cluster: not found
...lots of image.config.openshift.io/cluster: not found...
W0330 01:14:18.976550       1 observe_images.go:41] image.config.openshift.io/cluster: not found
I0330 01:14:20.082583       1 streamwatcher.go:103] Unexpected EOF during watch stream event decoding: unexpected EOF
...

The cluster-version operator seems pretty confused too:

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-4.0/6337/artifacts/e2e-aws/pods/openshift-cluster-version_cluster-version-operator-7b85fcb9fd-h7wnn_cluster-version-operator.log.gz | gunzip
I0330 01:00:32.868556       1 start.go:23] ClusterVersionOperator v3.11.0-252-gfc1c824a-dirty
I0330 01:00:32.868944       1 merged_client_builder.go:122] Using in-cluster configuration
I0330 01:00:32.871444       1 payload.go:206] Loading updatepayload from "/"
I0330 01:00:32.873658       1 decoder.go:224] decoding stream as YAML
...more YAML decoding...
I0330 01:00:32.991465       1 decoder.go:224] decoding stream as YAML
I0330 01:00:32.992512       1 leaderelection.go:187] attempting to acquire leader lease  openshift-cluster-version/version...
E0330 01:00:32.992973       1 leaderelection.go:252] error retrieving resource lock openshift-cluster-version/version: Get https://127.0.0.1:6443/api/v1/namespaces/openshift-cluster-version/configmaps/version: d
ial tcp 127.0.0.1:6443: connect: connection refused
I0330 01:00:32.992998       1 leaderelection.go:192] failed to acquire lease openshift-cluster-version/version
...more connection refusals...
E0330 01:25:21.068186       1 leaderelection.go:252] error retrieving resource lock openshift-cluster-version/version: Get https://127.0.0.1:6443/api/v1/namespaces/openshift-cluster-version/configmaps/version: dial tcp 127.0.0.1:6443: connect: connection refused
I0330 01:25:21.068222       1 leaderelection.go:192] failed to acquire lease openshift-cluster-version/version

Comment 2 Stefan Schimanski 2019-04-01 09:38:38 UTC
Any hint why the scheduler took so long to launch?

Comment 3 Michal Fojtik 2019-04-01 09:50:20 UTC
The connection refused and EOF might suggests there is something fishy going on in networking/SDN operators that causing disruption of traffic.

"image.config.openshift.io/cluster" is just log spam (we waiting for registry operator to create that CR)

"Unexpected EOF during watch stream event decoding: unexpected EOF" is what is suspicious... even if API server rolling new revision, it will shutdown gracefully and we won't see an unexpected EOF error.

Comment 4 Devan Goodwin 2019-04-02 11:16:49 UTC
Observed again today: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_cluster-kube-apiserver-operator/362/pull-ci-openshift-cluster-kube-apiserver-operator-master-e2e-aws/1849 

Appears to be identical to Trevor's original analysis, two rounds, first fails on timeout, second fails on connection refused.

Comment 5 Michal Fojtik 2019-04-05 08:02:11 UTC
kube-apiserver-operator-86bcf47c46-zrdm8                              Initialized=True, Ready=False, ContainersReady=False, PodScheduled=True
  [!] Container "operator" restarted 10 times, last exit 2 caused by:
   because it doesn't contain any IP SANs

goroutine 1 [running]:
github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerBuilder).Run(0xc420558090, 0xc42000e9e8, 0x1f1b4e0, 0xc4203c89c0, 0x3, 0x3)
                                                                      /go/src/github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/builder.go:175 +0xc2c
github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).StartController(0xc420397080, 0x1f1b520, 0xc420052068, 0x2e3a0f8, 0x0)
                                                                      /go/src/github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:193 +0x638
github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).NewCommand.func1(0xc4204b0a00, 0xc420579de0, 0x0, 0x2)
                                                                      /go/src/github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:68 +0x271
github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/spf13/cobra.(*Command).execute(0xc4204b0a00, 0xc420579d80, 0x2, 0x2, 0xc4204b0a00, 0xc420579d80)
                                                                      /go/src/github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4204b0780, 0x0, 0x0, 0x0)
                                                                      /go/src/github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/spf13/cobra/command.go:846 +0x30a
github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4204b0780, 0x1e0ad48, 0x2e17220)
                                                                      /go/src/github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/spf13/cobra/command.go:794 +0x2b
main.main()
                                                                      /go/src/github.com/openshift/cluster-kube-apiserver-operator/cmd/cluster-kube-apiserver-operator/main.go:37 +0x187


kube-controller-manager-operator-54b975ff6b-g98dn                     Initialized=True, Ready=False, ContainersReady=False, PodScheduled=True
  [!] Container "operator" restarted 10 times, last exit 2 caused by:
   library-go/pkg/controller/controllercmd.(*ControllerBuilder).Run(0xc420172240, 0xc42000e868, 0x1f03ba0, 0xc42044f780, 0x3, 0x3)
                                                                      /go/src/github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/builder.go:175 +0xc2c
github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).StartController(0xc42010a630, 0x1f03be0, 0xc420052038, 0x2e810e8, 0x0)
                                                                      /go/src/github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:193 +0x638
github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).NewCommand.func1(0xc420722280, 0xc4205943a0, 0x0, 0x2)
                                                                      /go/src/github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:68 +0x271
github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/spf13/cobra.(*Command).execute(0xc420722280, 0xc420594340, 0x2, 0x2, 0xc420722280, 0xc420594340)
                                                                      /go/src/github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420722000, 0xc420722500, 0xc420722280, 0xc420722c80)
                                                                      /go/src/github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/spf13/cobra/command.go:846 +0x30a
github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420722000, 0x1dfa4d8, 0x2e5e2c0)
                                                                      /go/src/github.com/openshift/cluster-kube-controller-manager-operator/vendor/github.com/spf13/cobra/command.go:794 +0x2b
main.main()
                                                                      /go/src/github.com/openshift/cluster-kube-controller-manager-operator/cmd/cluster-kube-controller-manager-operator/main.go:33 +0x187


openshift-kube-scheduler-operator-566f55669-lpmwb                     Initialized=True, Ready=False, ContainersReady=False, PodScheduled=True
  [!] Container "kube-scheduler-operator-container" restarted 10 times, last exit 2 caused by:
   any IP SANs

goroutine 1 [running]:
github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerBuilder).Run(0xc42040c510, 0xc4200d0348, 0x1ddd700, 0xc4204bc940, 0x3, 0x3)
                                                                      /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/builder.go:172 +0xc14
github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).StartController(0xc42009e000, 0x1ddd740, 0xc4200ce010, 0x2c9d2b8, 0x0)
                                                                      /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:193 +0x638
github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).NewCommand.func1(0xc42010c780, 0xc420754320, 0x0, 0x2)
                                                                      /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:68 +0x271
github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/spf13/cobra.(*Command).execute(0xc42010c780, 0xc4207542c0, 0x2, 0x2, 0xc42010c780, 0xc4207542c0)
                                                                      /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc42010c500, 0xc42010cf00, 0xc42010cc80, 0xc42010c780)
                                                                      /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/spf13/cobra/command.go:846 +0x30a
github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42010c500, 0x1ce7208, 0x2c7bcc0)
                                                                      /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/spf13/cobra/command.go:794 +0x2b
main.main()
                                                                      /go/src/github.com/openshift/cluster-kube-scheduler-operator/cmd/cluster-kube-scheduler-operator/main.go:32 +0x187

Comment 6 Michal Fojtik 2019-04-05 08:04:22 UTC
From the code:

vendor/github.com/google/certificate-transparency-go/x509/verify.go
111:			return "x509: cannot validate certificate for " + h.Host + " because it doesn't contain any IP SANs"

Comment 8 W. Trevor King 2019-04-05 08:28:37 UTC
I'm setting this blocked by 1694223, which is about improved log-gathering before bootstrap resource teardown.


Note You need to log in before you can comment on or make changes to this bug.