Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1692730 - Fail to upgrade ocp to non latest version
Summary: Fail to upgrade ocp to non latest version
Keywords:
Status: ASSIGNED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Upgrade
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.10.z
Assignee: Patrick Dillon
QA Contact: liujia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-26 10:22 UTC by liujia
Modified: 2019-04-04 16:02 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-03 19:23:28 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description liujia 2019-03-26 10:22:17 UTC
Description of problem:
Upgrade ocp v3.9 to v3.10 when specify openshift_pkg_version to an old version. Upgrade failed at TASK [Install openshift clients on first master] *******************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/verify_cluster.yml:47
Monday 25 March 2019  09:21:30 +0000 (0:00:00.035)       0:59:41.307 ********** 
FAILED - RETRYING: Install openshift clients on first master (3 retries left).
FAILED - RETRYING: Install openshift clients on first master (2 retries left).
FAILED - RETRYING: Install openshift clients on first master (1 retries left).
fatal: [x]: FAILED! => {"attempts": 3, "changed": false, "failed": true, "msg": "No package matching 'atomic-openshift-clients-3.10.127' found available, installed or updated", "rc": 126, "results": ["No package matching 'atomic-openshift-clients-3.10.127' found available, installed or updated"]}

Debug info:
1) The first cause should be excluder.
# yum search atomic-openshift-clients --showdup
Loaded plugins: amazon-id, search-disabled-repos
=================================================================== N/S matched: atomic-openshift-clients ===================================================================
atomic-openshift-clients-3.10.129-1.git.0.05be54b.el7.x86_64 : Origin Client binaries for Linux

  Name and summary matches only, use "search all" for everything.

Checked that the excluder still enabled, which is not consistant with the task [openshift_excluder : disable openshift excluder] before this failed task.

# cat /etc/yum.conf |grep exclude
exclude= atomic-openshift-tests  atomic-openshift-hyperkube  atomic-openshift-sdn-ovs  atomic-openshift-recycle  atomic-openshift-pod  atomic-openshift-node  atomic-openshift-master  atomic-openshift-clients-redistributable  atomic-openshift-clients  atomic-openshift  docker*1.20*  docker*1.19*  docker*1.18*  docker*1.17*  docker*1.16*  docker*1.15*  docker*1.14* 

Run unexclude manually, then "yum search" can list avialble atomic-openshift-clients pkg.

2) And another cause should be atomic-openshift-clients-3.10.129(latest version) has been installed in previous TASK [Install openshift clients on first master] as a dependency. 

# cat /etc/yum.conf |grep exclude
exclude= docker*1.20*  docker*1.19*  docker*1.18*  docker*1.17*  docker*1.16*  docker*1.15*  docker*1.14* 

# yum install atomic-openshift-clients-3.10.127
Loaded plugins: amazon-id, search-disabled-repos
aos                                                                                                                                                   | 1.3 kB  00:00:00     
aos310                                                                                                                                                | 1.3 kB  00:00:00     
aos_addon3_10                                                                                                                                         | 1.3 kB  00:00:00     
Package matching atomic-openshift-clients-3.10.127-1.git.0.dab74c6.el7.x86_64 already installed. Checking for update.
Nothing to do
 
# rpm -qa|grep clients
openssh-clients-7.4p1-16.el7.x86_64
atomic-openshift-clients-3.10.129-1.git.0.05be54b.el7.x86_64

So i dig more on a fresh installed v3.9 cluster. When enable both latest v3.10.129 and v3.10.127 repos. Try to install atomic-openshift-clients-3.10.127, the result is that atomic-openshift-clients-3.10.129 will be installed. This comply with the cause 2), which showed in upgrade log.

# yum install atomic-openshift-clients-3.10.127
Loaded plugins: amazon-id, search-disabled-repos
aos                                                                                                                                                   | 1.3 kB  00:00:00     
aos_addon3_10                                                                                                                                         | 1.3 kB  00:00:00     
oso-rhui-rhel-server-extras                                                                                                                           | 4.0 kB  00:00:00     
oso-rhui-rhel-server-fast-datapath                                                                                                                    | 4.0 kB  00:00:00     
oso-rhui-rhel-server-releases                                                                                                                         | 4.0 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package atomic-openshift-clients.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated
--> Processing Dependency: atomic-openshift-clients = 3.9.71-1.git.0.dba24d1.el7 for package: atomic-openshift-3.9.71-1.git.0.dba24d1.el7.x86_64
---> Package atomic-openshift-clients.x86_64 0:3.10.127-1.git.0.dab74c6.el7 will be an update
--> Running transaction check
---> Package atomic-openshift.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated
--> Processing Dependency: atomic-openshift = 3.9.71-1.git.0.dba24d1.el7 for package: atomic-openshift-node-3.9.71-1.git.0.dba24d1.el7.x86_64
--> Processing Dependency: atomic-openshift = 3.9.71-1.git.0.dba24d1.el7 for package: atomic-openshift-master-3.9.71-1.git.0.dba24d1.el7.x86_64
---> Package atomic-openshift.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update
--> Processing Dependency: atomic-openshift-clients = 3.10.129-1.git.0.05be54b.el7 for package: atomic-openshift-3.10.129-1.git.0.05be54b.el7.x86_64
aos/filelists                                                                                                                                         | 665 kB  00:00:00     
--> Running transaction check
---> Package atomic-openshift-clients.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated
---> Package atomic-openshift-clients.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated
---> Package atomic-openshift-clients.x86_64 0:3.10.127-1.git.0.dab74c6.el7 will be an update
---> Package atomic-openshift-clients.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update
---> Package atomic-openshift-master.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated
---> Package atomic-openshift-master.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update
---> Package atomic-openshift-node.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated
--> Processing Dependency: atomic-openshift-node = 3.9.71-1.git.0.dba24d1.el7 for package: atomic-openshift-sdn-ovs-3.9.71-1.git.0.dba24d1.el7.x86_64
---> Package atomic-openshift-node.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update
--> Processing Dependency: atomic-openshift-hyperkube = 3.10.129-1.git.0.05be54b.el7 for package: atomic-openshift-node-3.10.129-1.git.0.05be54b.el7.x86_64
--> Running transaction check
---> Package atomic-openshift-hyperkube.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be installed
---> Package atomic-openshift-sdn-ovs.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated
---> Package atomic-openshift-sdn-ovs.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================
 Package                                          Arch                         Version                                             Repository                           Size
=============================================================================================================================================================================
Installing for dependencies:
 atomic-openshift-hyperkube                       x86_64                       3.10.129-1.git.0.05be54b.el7                        aos_addon3_10                        33 M
Updating for dependencies:
 atomic-openshift                                 x86_64                       3.10.129-1.git.0.05be54b.el7                        aos_addon3_10                        39 M
 atomic-openshift-clients                         x86_64                       3.10.129-1.git.0.05be54b.el7                        aos_addon3_10                        20 M
 atomic-openshift-master                          x86_64                       3.10.129-1.git.0.05be54b.el7                        aos_addon3_10                       958 k
 atomic-openshift-node                            x86_64                       3.10.129-1.git.0.05be54b.el7                        aos_addon3_10                       5.2 M
 atomic-openshift-sdn-ovs                         x86_64                       3.10.129-1.git.0.05be54b.el7                        aos_addon3_10                       4.6 M

Transaction Summary
=============================================================================================================================================================================
Install             ( 1 Dependent package)
Upgrade             ( 5 Dependent packages)

Total download size: 103 M
Is this ok [y/d/N]: 



Version-Release number of the following components:
ansible-2.4.6.0-1.el7ae.noarch
openshift-ansible-3.10.127-1.git.0.131da09.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. Install ocp v3.9
2. Enable both v3.10.127 and latest v3.10.129 repos on the hosts
3. Specify 
openshift_release=v3.10
openshift_pkg_version=-3.10.127
openshift_image_tag=v3.10.127
4. Run upgrade
/usr/share/ansible/openshift-ansible/playbooks/openshift-master/openshift_node_group.yml
/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade.yml

Actual results:
Upgrade failed.

Expected results:
Ocp was upgraded to v3.10.127.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 5 Patrick Dillon 2019-04-03 19:23:28 UTC
> 2. Enable both v3.10.127 and latest v3.10.129 repos on the hosts

The target host should only have the repos enabled for the appropriate version (in this case, disable v3.10.129). I don't believe this is a bug. Please reopen if you think otherwise or something develops.

Comment 6 liujia 2019-04-04 01:55:48 UTC
(In reply to Patrick Dillon from comment #5)
> > 2. Enable both v3.10.127 and latest v3.10.129 repos on the hosts
> 
> The target host should only have the repos enabled for the appropriate
> version (in this case, disable v3.10.129). I don't believe this is a bug.
> Please reopen if you think otherwise or something develops.

Hi Patrick

The key point for the issue is not how many repos enabled, but multi versions are avialable during upgrade. QE enabled two repos is just to simulate two pre-release versions avaialable. In customer env, even only one ose z-stream repo enabled in target host, but there will be many versions avialable in this repo, such like[1](there are v3.10.111,v3.10.119, and so on in one repo v3.10). So when they do upgrade to a non-latest version, they will hit the issue.

# yum install atomic-openshift-clients-3.10.111
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
aos39-devel-install                                                                                                                                   | 4.0 kB  00:00:00     
fast-datapath                                                                                                                                         | 4.0 kB  00:00:00     
rhel7                                                                                                                                                 | 3.4 kB  00:00:00     
rhel7-extra                                                                                                                                           | 3.4 kB  00:00:00     
(1/12): aos39-devel-install/group                                                                                                                     |  124 B  00:00:00     
(2/12): aos39-devel-install/updateinfo                                                                                                                |  83 kB  00:00:00     
(3/12): fast-datapath/updateinfo                                                                                                                      |  22 kB  00:00:00     
(4/12): fast-datapath/group                                                                                                                           |  124 B  00:00:00     
(5/12): aos39-devel-install/primary_db                                                                                                                | 300 kB  00:00:00     
(6/12): fast-datapath/primary_db                                                                                                                      | 108 kB  00:00:00     
(7/12): rhel7/group                                                                                                                                   | 774 kB  00:00:00     
(8/12): rhel7-extra/group                                                                                                                             |  124 B  00:00:00     
(9/12): rhel7-extra/updateinfo                                                                                                                        | 169 kB  00:00:00     
(10/12): rhel7-extra/primary_db                                                                                                                       | 519 kB  00:00:00     
(11/12): rhel7/updateinfo                                                                                                                             | 3.0 MB  00:00:01     
(12/12): rhel7/primary_db                                                                                                                             |  54 MB  00:00:28     
Resolving Dependencies
--> Running transaction check
---> Package atomic-openshift-clients.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated
--> Processing Dependency: atomic-openshift-clients = 3.9.74-1.git.0.78e56ea.el7 for package: atomic-openshift-3.9.74-1.git.0.78e56ea.el7.x86_64
---> Package atomic-openshift-clients.x86_64 0:3.10.111-1.git.0.84b439a.el7 will be an update
--> Running transaction check
---> Package atomic-openshift.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated
--> Processing Dependency: atomic-openshift = 3.9.74-1.git.0.78e56ea.el7 for package: atomic-openshift-node-3.9.74-1.git.0.78e56ea.el7.x86_64
--> Processing Dependency: atomic-openshift = 3.9.74-1.git.0.78e56ea.el7 for package: atomic-openshift-master-3.9.74-1.git.0.78e56ea.el7.x86_64
---> Package atomic-openshift.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update
--> Processing Dependency: atomic-openshift-clients = 3.10.119-1.git.0.a1cf669.el7 for package: atomic-openshift-3.10.119-1.git.0.a1cf669.el7.x86_64
aos39-devel-install/filelists_db                                                                                                                      | 987 kB  00:00:00     
--> Running transaction check
---> Package atomic-openshift-clients.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated
---> Package atomic-openshift-clients.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated
---> Package atomic-openshift-clients.x86_64 0:3.10.111-1.git.0.84b439a.el7 will be an update
---> Package atomic-openshift-clients.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update
---> Package atomic-openshift-master.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated
---> Package atomic-openshift-master.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update
---> Package atomic-openshift-node.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated
--> Processing Dependency: atomic-openshift-node = 3.9.74-1.git.0.78e56ea.el7 for package: atomic-openshift-sdn-ovs-3.9.74-1.git.0.78e56ea.el7.x86_64
---> Package atomic-openshift-node.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update
--> Processing Dependency: atomic-openshift-hyperkube = 3.10.119-1.git.0.a1cf669.el7 for package: atomic-openshift-node-3.10.119-1.git.0.a1cf669.el7.x86_64
--> Running transaction check
---> Package atomic-openshift-hyperkube.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be installed
---> Package atomic-openshift-sdn-ovs.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated
---> Package atomic-openshift-sdn-ovs.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================
 Package                                        Arch                       Version                                             Repository                               Size
=============================================================================================================================================================================
Installing for dependencies:
 atomic-openshift-hyperkube                     x86_64                     3.10.119-1.git.0.a1cf669.el7                        aos39-devel-install                      33 M
Updating for dependencies:
 atomic-openshift                               x86_64                     3.10.119-1.git.0.a1cf669.el7                        aos39-devel-install                      39 M
 atomic-openshift-clients                       x86_64                     3.10.119-1.git.0.a1cf669.el7                        aos39-devel-install                      20 M
 atomic-openshift-master                        x86_64                     3.10.119-1.git.0.a1cf669.el7                        aos39-devel-install                     958 k
 atomic-openshift-node                          x86_64                     3.10.119-1.git.0.a1cf669.el7                        aos39-devel-install                     5.2 M
 atomic-openshift-sdn-ovs                       x86_64                     3.10.119-1.git.0.a1cf669.el7                        aos39-devel-install                     4.6 M

Transaction Summary
=============================================================================================================================================================================
Install             ( 1 Dependent package)
Upgrade             ( 5 Dependent packages)

Total download size: 103 M
Is this ok [y/d/N]: 


So open the bug again.

[1] http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ose/3.10/os/Packages/a/

Comment 7 Russell Teague 2019-04-04 16:02:28 UTC
Patrick,
It looks like this is possibly a regression caused by https://github.com/openshift/openshift-ansible/pull/10783 in openshift-ansible-3.10.100-1 and later.

There is a circular RPM dependency which causes all packages to get upgraded in task [Install openshift clients on first master].  During upgrades of these packages we must always specify the version of all dependent packages to prevent this circular dependency and is what is done during the regular package upgrade tasks.  We need to revisit the fix which introduced this issue to see if we can work around needing to install all dependencies or do the check at a later time when binaries are properly updated.


Note You need to log in before you can comment on or make changes to this bug.