Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1690816 - Progress percentage calculation should be always increasing during upgrading
Summary: Progress percentage calculation should be always increasing during upgrading
Keywords:
Status: NEW
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Upgrade
Version: 4.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.2.0
Assignee: Abhinav Dahiya
QA Contact: liujia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-20 09:56 UTC by weiwei jiang
Modified: 2019-04-01 19:44 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description weiwei jiang 2019-03-20 09:56:49 UTC
Description of problem:
During upgrading, found the progress reported in clusterversion is not increasing.
[root@preserved-bind-and-bastion ~]# oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-19-220030   True        True          8m51s   Working towards 4.0.0-0.nightly-2019-03-19-220030: 33% complete                                                                                                                                                                                              
[root@preserved-bind-and-bastion ~]# oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-19-220030   True        True          12m     Working towards 4.0.0-0.nightly-2019-03-19-220030: 46% complete
[root@preserved-bind-and-bastion ~]# oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-19-220030   True        True          15m     Working towards 4.0.0-0.nightly-2019-03-19-220030: 2% complete
[root@preserved-bind-and-bastion ~]# oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-19-220030   True        True          16m     Working towards 4.0.0-0.nightly-2019-03-19-220030: 2% complete
[root@preserved-bind-and-bastion ~]# oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-19-220030   True        True          16m     Working towards 4.0.0-0.nightly-2019-03-19-220030: 2% complete


Version-Release number of the following components:
Upgrade from 4.0.0-0.nightly-2019-03-19-004004 to 4.0.0-0.nightly-2019-03-19-220030

How reproducible:
Always

Steps to Reproduce:
1. Install version 4.0.0-0.nightly-2019-03-19-004004
2. Upgrade it to 4.0.0-0.nightly-2019-03-19-220030
3. Check progress percentage during upgrading

Actual results:
progress percentage is not increasing

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 weiwei jiang 2019-03-22 02:13:28 UTC
Recheck this and this means there are 2 round upgrade happen here? so the percentage is re-calculated?

version   4.0.0-0.nightly-2019-03-22-002648   True        True          17m     Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          17m     Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          17m     Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          17m     Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          17m     Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          17m     Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          17m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          17m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          17m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          18m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          18m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          18m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          18m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          18m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          18m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          18m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          18m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          19m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          19m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          19m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          19m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          19m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          19m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          19m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          19m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          19m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          20m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          20m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          20m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          20m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          20m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          20m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          20m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          20m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          20m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          21m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          21m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          21m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
The connection to the server api.wjiang-ocp.qe.devcluster.openshift.com:6443 was refused - did you specify the right host or port?
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          21m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
Error from server (NotFound): Unable to list {"config.openshift.io" "v1" "clusterversions"}: the server could not find the requested resource (get clusterversions.config.openshift.io)
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          21m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          21m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          22m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          23m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          23m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          23m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          23m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          23m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          23m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          23m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          23m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          23m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          24m     Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          24m     Working towards 4.0.0-0.nightly-2019-03-22-002648: 2% complete
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-03-22-002648   True        True          24m     Working towards 4.0.0-0.nightly-2019-03-22-002648: 2% complete

Comment 2 W. Trevor King 2019-03-28 23:00:37 UTC
This is a cluster-version-operator issue, not an installer issue, so I'm reassigning to the Upgrade component.

Comment 3 W. Trevor King 2019-04-01 19:44:35 UTC
Colin points out that OS upgrades [1] can lead to node restarts, which could force cluster-version operator moves (as the node it was running on is drained for the upgrade), which will result in a new sync cycle (as the new CVO pod comes up somewhere else).  So if we want something closer to monotonic progress reports, we'd need each CVO sync cycle to take a look at the existing object state and silently fast-forward through tasks which were already in sync.

[1]: https://github.com/openshift/machine-config-operator/blob/2b28eb287e5bc7b654680a6e85f767ff05604371/docs/OSUpgrades.md


Note You need to log in before you can comment on or make changes to this bug.