Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1518662 - OSP11 -> OSP12 upgrade: pre-upgrade validations are preventing a re-run of the upgrade-non-controller.sh script to upgrade a compute node after a failed attempt
Summary: OSP11 -> OSP12 upgrade: pre-upgrade validations are preventing a re-run of th...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: z3
: 12.0 (Pike)
Assignee: Marios Andreou
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-29 12:34 UTC by Marius Cornea
Modified: 2018-08-20 12:59 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-common-7.6.9-1.el7ost
Doc Type: Bug Fix
Doc Text:
Additional non-controller upgrade attempts after a failed upgrade can fail during service validation if services are not running. To prevent such upgrade failures you can skip services validation. Pass the option "--skip-tags validation" to the Ansible invocation. For example: upgrade-non-controller.sh --upgrade compute-0 --ansible-opts "--skip-tags validation"
Clone Of:
Environment:
Last Closed: 2018-08-20 12:58:39 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1735731 None None None 2017-12-04 08:34:22 UTC
Red Hat Product Errata RHSA-2018:2331 None None None 2018-08-20 12:59:20 UTC
OpenStack gerrit 526047 None None None 2017-12-06 13:41:04 UTC

Description Marius Cornea 2017-11-29 12:34:00 UTC
Description of problem:
OSP11 -> OSP12 upgrade: unable to re-run the upgrade-non-controller.sh script to upgrade a compute node after a failed attempt because the pre-upgrade validations are failing.

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.3-16.el7ost.noarch
openstack-tripleo-common-7.6.3-6.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11
2. Upgrade to OSP12
3. Complete major-upgrade-composable-steps-docker.yaml step successfuly
4. Run upgrade-non-controller.sh --upgrade compute-0 which fails due to unreacheable repositories:

TASK [Upgrade os-net-config] ******************************************************************************************************************************************************************************************************************
fatal: [192.168.24.11]: FAILED! => {"changed": true, "failed": true, "msg": "http://rhos-qe-mirror-brq.usersys.redhat.com/rcm-guest/puddles/OpenStack/12.0-RHEL-7/latest/RH7-RHOS-12.0/x86_64/os/Packages/python2-pbr-3.1.1-1.el7ost.noarch.rpm: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=rhelosp-12.0-puddle clean metadata\nTrying other mirror.\nhttp://rhos-qe-mirror-brq.usersys.redhat.com/rcm-guest/puddles/OpenStack/12.0-RHEL-7/latest/RH7-RHOS-12.0/x86_64/os/Packages/os-net-config-7.3.1-1.el7ost.noarch.rpm: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=rhelosp-12.0-puddle clean metadata\nTrying other mirror.\n\n\nError downloading packages:\n  os-net-config-7.3.1-1.el7ost.noarch: [Errno 256] No more mirrors to try.\n  python2-pbr-3.1.1-1.el7ost.noarch: [Errno 256] No more mirrors to try.\n\n", "rc": 1, "results": ["Loaded plugins: product-id, search-disabled-repos, subscription-manager\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\nResolving Dependencies\n--> Running transaction check\n---> Package os-net-config.noarch 0:6.1.0-2.el7ost will be updated\n---> Package os-net-config.noarch 0:7.3.1-1.el7ost will be an update\n--> Processing Dependency: python-pbr >= 2.0.0 for package: os-net-config-7.3.1-1.el7ost.noarch\n--> Running transaction check\n---> Package python-pbr.noarch 0:1.10.0-2.el7ost will be obsoleted\n---> Package python2-pbr.noarch 0:3.1.1-1.el7ost will be obsoleting\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package           Arch       Version             Repository               Size\n================================================================================\nInstalling:\n python2-pbr       noarch     3.1.1-1.el7ost      rhelosp-12.0-puddle     191 k\n     replacing  python-pbr.noarch 1.10.0-2.el7ost\nUpdating:\n os-net-config     noarch     7.3.1-1.el7ost      rhelosp-12.0-puddle     260 k\n\nTransaction Summary\n================================================================================\nInstall  1 Package\nUpgrade  1 Package\n\nTotal download size: 451 k\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n"]}

5. Fix repositories issue

6. Re-run upgrade-non-controller.sh --upgrade compute-0

Actual results:
The command fails the pre-upgrade validation check for the neutron-openvswitch-agent service:

TASK [PreUpgrade step0,validation: Check service neutron-openvswitch-agent is running] ********************************************************************************************************************************************************
fatal: [192.168.24.11]: FAILED! => {"changed": true, "cmd": "/usr/bin/systemctl show 'neutron-openvswitch-agent' --property ActiveState | grep '\\bactive\\b'", "delta": "0:00:00.008534", "end": "2017-11-29 12:26:33.244232", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-11-29 12:26:33.235698", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}


Expected results:
We should have a way to disable to pre-upgrade validation checks for the upgrade-non-controller script to allow re-running it after a failed attempt.

Additional info:

Workaround: ssh to the compute node and manually start the services which were stopped during the failed upgrade attempt:

systemctl start neutron-openvswitch-agent

Then re-run the upgrade-non-controller.sh script.

Comment 1 Carlos Camacho 2017-11-29 14:39:55 UTC
Yes, we can not pass --skip-validations to upgrade-non-controller.sh

Comment 2 Marios Andreou 2017-11-30 10:47:45 UTC
taking for investigation as we discussed on scrum last night thanks

Comment 3 Marios Andreou 2017-11-30 16:32:06 UTC
fwiw I don't think this is a blocker for 12 and would be nice to get into the first async release please. Upstream patch posted today needs some more testing

Comment 6 Lon Hohberger 2018-03-29 10:34:32 UTC
According to our records, this should be resolved by openstack-tripleo-common-7.6.9-3.el7ost.  This build is available now.

Comment 8 Yurii Prokulevych 2018-07-02 14:31:39 UTC
Verified with openstack-tripleo-common-7.6.13-1.el7ost.noarch

upgrade-non-controller.sh --upgrade compute-0 -O '--skip-tags validation'

Comment 11 errata-xmlrpc 2018-08-20 12:58:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2331


Note You need to log in before you can comment on or make changes to this bug.