Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1684431 - KubeVersionMismatch: There are 2 different versions of Kubernetes components running in Alerts
Summary: KubeVersionMismatch: There are 2 different versions of Kubernetes components ...
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.1.0
Assignee: Frederic Branczyk
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-01 09:18 UTC by Neeraj
Modified: 2019-03-28 01:22 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Neeraj 2019-03-01 09:18:49 UTC
Description of problem:
There is an Alert in the OpenShift console that reads "KubeVersionMismatch: There are 2 different versions of Kubernetes components running.". This was just after installation and without any customisation from my side.

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:

```
$ openshift-install version
openshift-install v0.12.0
$ openshift-install --dir=taakrsi4-demo3/ create cluster
```

Prometheus (query "kubernetes_build_info{job!="kube-dns"}") reports:

```
kubernetes_build_info{buildDate="2019-01-29T23:36:59Z",compiler="gc",endpoint="https",gitCommit="f39ab668d3",gitTreeState="clean",gitVersion="v1.12.4+f39ab668d3",goVersion="go1.10.3",instance="10.0.27.179:10251",job="scheduler",major="1",minor="12+",namespace="openshift-kube-scheduler",platform="linux/amd64",pod="openshift-kube-scheduler-ip-10-0-27-179.eu-central-1.compute.internal",service="scheduler"}	1
kubernetes_build_info{buildDate="2019-01-29T23:36:59Z",compiler="gc",endpoint="https",gitCommit="f39ab668d3",gitTreeState="clean",gitVersion="v1.12.4+f39ab668d3",goVersion="go1.10.3",instance="10.0.27.179:10252",job="kube-controller-manager",major="1",minor="12+",namespace="openshift-kube-controller-manager",platform="linux/amd64",pod="openshift-kube-controller-manager-ip-10-0-27-179.eu-central-1.compute.internal",service="kube-controller-manager"}	1
kubernetes_build_info{buildDate="2019-01-29T23:36:59Z",compiler="gc",endpoint="https",gitCommit="f39ab668d3",gitTreeState="clean",gitVersion="v1.12.4+f39ab668d3",goVersion="go1.10.3",instance="10.0.47.122:10251",job="scheduler",major="1",minor="12+",namespace="openshift-kube-scheduler",platform="linux/amd64",pod="openshift-kube-scheduler-ip-10-0-47-122.eu-central-1.compute.internal",service="scheduler"}	1
kubernetes_build_info{buildDate="2019-01-29T23:36:59Z",compiler="gc",endpoint="https",gitCommit="f39ab668d3",gitTreeState="clean",gitVersion="v1.12.4+f39ab668d3",goVersion="go1.10.3",instance="10.0.47.122:10252",job="kube-controller-manager",major="1",minor="12+",namespace="openshift-kube-controller-manager",platform="linux/amd64",pod="openshift-kube-controller-manager-ip-10-0-47-122.eu-central-1.compute.internal",service="kube-controller-manager"}	1
kubernetes_build_info{buildDate="2019-01-29T23:36:59Z",compiler="gc",endpoint="https",gitCommit="f39ab668d3",gitTreeState="clean",gitVersion="v1.12.4+f39ab668d3",goVersion="go1.10.3",instance="10.0.9.56:10251",job="scheduler",major="1",minor="12+",namespace="openshift-kube-scheduler",platform="linux/amd64",pod="openshift-kube-scheduler-ip-10-0-9-56.eu-central-1.compute.internal",service="scheduler"}	1
kubernetes_build_info{buildDate="2019-01-29T23:36:59Z",compiler="gc",endpoint="https",gitCommit="f39ab668d3",gitTreeState="clean",gitVersion="v1.12.4+f39ab668d3",goVersion="go1.10.3",instance="10.0.9.56:10252",job="kube-controller-manager",major="1",minor="12+",namespace="openshift-kube-controller-manager",platform="linux/amd64",pod="openshift-kube-controller-manager-ip-10-0-9-56.eu-central-1.compute.internal",service="kube-controller-manager"}	1
kubernetes_build_info{buildDate="2019-02-01T23:06:11Z",compiler="gc",endpoint="https-metrics",gitCommit="91d94b671b",gitTreeState="clean",gitVersion="v1.12.4+91d94b671b",goVersion="go1.10.3",instance="10.0.135.74:10250",job="kubelet",major="1",minor="12+",namespace="kube-system",node="ip-10-0-135-74.eu-central-1.compute.internal",platform="linux/amd64",service="kubelet"}	1
kubernetes_build_info{buildDate="2019-02-01T23:06:11Z",compiler="gc",endpoint="https-metrics",gitCommit="91d94b671b",gitTreeState="clean",gitVersion="v1.12.4+91d94b671b",goVersion="go1.10.3",instance="10.0.145.112:10250",job="kubelet",major="1",minor="12+",namespace="kube-system",node="ip-10-0-145-112.eu-central-1.compute.internal",platform="linux/amd64",service="kubelet"}	1
kubernetes_build_info{buildDate="2019-02-01T23:06:11Z",compiler="gc",endpoint="https-metrics",gitCommit="91d94b671b",gitTreeState="clean",gitVersion="v1.12.4+91d94b671b",goVersion="go1.10.3",instance="10.0.161.79:10250",job="kubelet",major="1",minor="12+",namespace="kube-system",node="ip-10-0-161-79.eu-central-1.compute.internal",platform="linux/amd64",service="kubelet"}	1
kubernetes_build_info{buildDate="2019-02-01T23:06:11Z",compiler="gc",endpoint="https-metrics",gitCommit="91d94b671b",gitTreeState="clean",gitVersion="v1.12.4+91d94b671b",goVersion="go1.10.3",instance="10.0.27.179:10250",job="kubelet",major="1",minor="12+",namespace="kube-system",node="ip-10-0-27-179.eu-central-1.compute.internal",platform="linux/amd64",service="kubelet"}	1
kubernetes_build_info{buildDate="2019-02-01T23:06:11Z",compiler="gc",endpoint="https-metrics",gitCommit="91d94b671b",gitTreeState="clean",gitVersion="v1.12.4+91d94b671b",goVersion="go1.10.3",instance="10.0.47.122:10250",job="kubelet",major="1",minor="12+",namespace="kube-system",node="ip-10-0-47-122.eu-central-1.compute.internal",platform="linux/amd64",service="kubelet"}	1
kubernetes_build_info{buildDate="2019-02-01T23:06:11Z",compiler="gc",endpoint="https-metrics",gitCommit="91d94b671b",gitTreeState="clean",gitVersion="v1.12.4+91d94b671b",goVersion="go1.10.3",instance="10.0.9.56:10250",job="kubelet",major="1",minor="12+",namespace="kube-system",node="ip-10-0-9-56.eu-central-1.compute.internal",platform="linux/amd64",service="kubelet"}
```
Actual results:

There are lot of waring of kubeversion mismatch due to different git Versions

Expected results:

A newly installed cluster does not show any open alerts out-of-the-box

Comment 1 Junqi Zhao 2019-03-04 03:10:43 UTC
Yes, it had this issue before, but should not be a monitoring issue, we just throw out the alerts, anyway, there is not such issue with
4.0.0-0.nightly-2019-02-28-054829

The payload you use maybe too old
check via
# oc get clusterversion

Comment 3 lserven 2019-03-27 14:50:04 UTC
We should be resolved as of February 7th thanks to https://github.com/openshift/cluster-monitoring-operator/pull/235. If we are still seeing this issue today, then it is a bug in the alerting rule. Junqi, can you please verify if the alert is still showing today?

-Lucas


Note You need to log in before you can comment on or make changes to this bug.