Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1696387 - Openshift-apiserver is failing to upgrade
Summary: Openshift-apiserver is failing to upgrade
Keywords:
Status: CLOSED DUPLICATE of bug 1694226
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 4.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.1.0
Assignee: Michal Fojtik
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-04 17:49 UTC by Abhinav Dahiya
Modified: 2019-04-11 12:05 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-09 11:20:29 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin pull 22425 None None None 2019-04-05 08:06:40 UTC

Description Abhinav Dahiya 2019-04-04 17:49:58 UTC
3 instances of openshift-apiserver failing

https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/908
https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/907
https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/905
https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/904

with errors like
```
    {
      "lastTransitionTime": "2019-04-04T11:20:38Z",
      "message": "Available: v1.quota.openshift.io is not ready: 503",
      "reason": "Available",
      "status": "False",
      "type": "Available"
    },

```
or
```
    {
      "lastTransitionTime": "2019-04-04T09:31:00Z",
      "message": "Available: v1.image.openshift.io is not ready: 500\nAvailable: v1.route.openshift.io is not ready: 500\nAvailable: v1.security.openshift.io is not ready: 500",
      "reason": "AvailableMultiple",
      "status": "False",
      "type": "Available"
    },

```

Comment 1 Abhinav Dahiya 2019-04-04 22:37:48 UTC
Another CI runs that saw this failure:

https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/928
https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/927
https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/926

Same error example:
curl https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/928/artifacts/e2e-aws-upgrade/clusteroperators.json | jq '.items[] | select(.status.conditions[] | .type == "Available" and .status != "True") | [.metadata.name, .status.conditions]'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 61288  100 61288    0     0   123k      0 --:--:-- --:--:-- --:--:--  123k
[
  "openshift-apiserver",
  [
    {
      "lastTransitionTime": "2019-04-04T20:43:13Z",
      "reason": "AsExpected",
      "status": "False",
      "type": "Failing"
    },
    {
      "lastTransitionTime": "2019-04-04T20:43:13Z",
      "reason": "AsExpected",
      "status": "False",
      "type": "Progressing"
    },
    {
      "lastTransitionTime": "2019-04-04T21:01:33Z",
      "message": "Available: v1.quota.openshift.io is not ready: 503",
      "reason": "Available",
      "status": "False",
      "type": "Available"
    },
    {
      "lastTransitionTime": "2019-04-04T20:43:13Z",
      "reason": "AsExpected",
      "status": "True",
      "type": "Upgradeable"
    }
  ]
]

Comment 2 Michal Fojtik 2019-04-05 07:53:24 UTC
This is kind of known. I suspect https://github.com/openshift/cluster-config-operator/pull/25 started this.
I need to merge https://github.com/openshift/origin/pull/22425 and this failure should go away.

Comment 3 Xingxing Xia 2019-04-08 03:29:01 UTC
In fresh env of latest payload 4.0.0-0.nightly-2019-04-05-165550, also see: 
oc get co 
... 
openshift-apiserver                  4.0.0-0.nightly-2019-04-05-165550   False ... 
... 
 
oc get co openshift-apiserver -o yaml 
... 
status: 
  conditions: 
  - lastTransitionTime: 2019-04-08T02:41:44Z 
    reason: AsExpected 
    status: "False" 
    type: Failing 
  - lastTransitionTime: 2019-04-08T02:41:00Z 
    reason: AsExpected 
    status: "False" 
    type: Progressing 
  - lastTransitionTime: 2019-04-08T03:11:03Z 
    message: 'Available: v1.quota.openshift.io is not ready: 503' 
    reason: Available 
    status: "False" 
    type: Available 
... 
 
oc get openshiftapiserver -o yaml 
... 
  status: 
    conditions: 
    - lastTransitionTime: 2019-04-08T02:32:47Z 
      reason: NoUnsupportedConfigOverrides 
      status: "True" 
      type: UnsupportedConfigOverridesUpgradeable 
    - lastTransitionTime: 2019-04-08T02:32:48Z 
      status: "False" 
      type: ConfigObservationFailing 
    - lastTransitionTime: 2019-04-08T02:36:32Z 
      status: "False" 
      type: ResourceSyncControllerFailing 
    - lastTransitionTime: 2019-04-08T03:26:32Z 
      message: 'v1.quota.openshift.io is not ready: 503' 
      status: "False" 
      type: Available 
...

Comment 4 Xingxing Xia 2019-04-08 07:24:04 UTC
Continuing comment 3:
$ watch oc get clusteroperator/openshift-apiserver
openshift-apiserver   4.0.0-0.nightly-2019-04-05-165550   False       False         False     0s
...
openshift-apiserver   4.0.0-0.nightly-2019-04-05-165550   True       False         False     0s

openshift-apiserver is always flapping between True and False

Comment 5 Michal Fojtik 2019-04-09 11:20:29 UTC

*** This bug has been marked as a duplicate of bug 1694226 ***


Note You need to log in before you can comment on or make changes to this bug.