Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1514722

Summary: yumdownloader fails to upgrade OVS if files exist in OVS_UPGRADE
Product: Red Hat OpenStack Reporter: Emilien Macchi <emacchi>
Component: openstack-tripleo-heat-templatesAssignee: Marios Andreou <mandreou>
Status: CLOSED CURRENTRELEASE QA Contact: Yurii Prokulevych <yprokule>
Severity: high Docs Contact:
Priority: medium    
Version: 9.0 (Mitaka)CC: augol, dbecker, emacchi, jamsmith, mandreou, mburns, morazi, rhel-osp-director-maint, sbandyop, slinaber
Target Milestone: asyncKeywords: TestOnly, Triaged, ZStream
Target Release: 9.0 (Mitaka)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-2.0.0-59.el7ost Doc Type: Bug Fix
Doc Text:
The openvswitch upgrade code in RHEL OSP Director has been improved to handle the re-running of failed upgrades. Prior to this update, data from a failed upgrade that included an opevswitch upgrade was not properly erased, and the upgrade re-run could fail with an error such as "error: openvswitch-2.5.0-15.git20160727.el7ost.x86_64: erase failed." This fix prevents ovs-upgrade related failures during a minor update of OSP9 or major upgrade to OSP9. See https://review.openstack.org/#/c/413524/ for more information.
Story Points: ---
Clone Of:
: 1558563 (view as bug list) Environment:
Last Closed: 2018-10-04 10:36:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1558563    

Description Emilien Macchi 2017-11-18 02:31:18 UTC
Description of problem:
When upgrading from OSP8 to OSP9, it upgrades openvswitch in a specific directory.
If the upgrade process fail and we want to restart it in a specific step, the upgrade scripts will fail because OVS_UPGRADE contains some packages and it will fail to erase it:

[heat-admin@controller-0 ~]$ sudo journalctl -u os-collect-config | grep -i openvswitch | tail -n 6Nov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: nsaction check\n---> Package openvswitch.x86_64 0:2.5.0-15.git20160727.el7ost will be reinstalled\n--> Finished Dependency Resolution\nUpdating openvswitch with nopostun option\n", "deploy_stderr": "mkdir: cannot create directory \u2018OVS_UPGRADE\u2019: File exists\nerror: openvswitch-2.5.0-15.git20160727.el7ost.x86_64: erase failed\n", "deploy_status_code": 2}Nov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: Manual upgrade of openvswitch - restart in postun detectedNov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: Attempting to downloading latest openvswitch with yumdownloaderNov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: ---> Package openvswitch.x86_64 0:2.5.0-15.git20160727.el7ost will be reinstalledNov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: Updating openvswitch with nopostun optionNov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: error: openvswitch-2.5.0-15.git20160727.el7ost.x86_64: erase failed
[heat-admin@ocd03-controller-0 ~]$ sudo journalctl -u os-collect-config | grep -i openvswitch | tail -n 6Nov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: nsaction check\n---> Package openvswitch.x86_64 0:2.5.0-15.git20160727.el7ost will be reinstalled\n--> Finished Dependency Resolution\nUpdating openvswitch with nopostun option\n", "deploy_stderr": "mkdir: cannot create directory \u2018OVS_UPGRADE\u2019: File exists\nerror: openvswitch-2.5.0-15.git20160727.el7ost.x86_64: erase failed\n", "deploy_status_code": 2}Nov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: Manual upgrade of openvswitch - restart in postun detectedNov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: Attempting to downloading latest openvswitch with yumdownloaderNov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: ---> Package openvswitch.x86_64 0:2.5.0-15.git20160727.el7ost will be reinstalledNov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: Updating openvswitch with nopostun optionNov 17 18:00:53 ocd03-controller-0.localdomain os-collect-config[1051]: error: openvswitch-2.5.0-15.git20160727.el7ost.x86_64: erase failed

We should handle this case and eventually cleanup OVS_UPGRADE directory before running "yumdownloader" command.

Comment 2 Shatadru Bandyopadhyay 2017-11-18 03:27:07 UTC
I was comparing the code is OSP 8 and later versions

From OSP 8 
~~~
/usr/share/openstack-tripleo-heat-templates/liberty/extraconfig
~~~
 46 # Special-case OVS for https://bugs.launchpad.net/tripleo/+bug/1635205
 47 if [[ -n $(rpm -q --scripts openvswitch | awk '/postuninstall/,/*/' | grep "systemctl.*try-restart") ]]; then
 48     echo "Manual upgrade of openvswitch - restart in postun detected"
 49     mkdir OVS_UPGRADE || true
 50     pushd OVS_UPGRADE
 51     echo "Attempting to downloading latest openvswitch with yumdownloader"
 52     yumdownloader --resolve openvswitch
 53     echo "Updating openvswitch with nopostun option"
 54     rpm -U --replacepkgs --nopostun ./*.rpm
 55     popd
 56 else
 57     echo "Skipping manual upgrade of openvswitch - no restart in postun detected"
 58 fi
~~~

In OSP 9 we run a cleanup :
~~~
     64 # Special-case OVS for https://bugs.launchpad.net/tripleo/+bug/1635205
     65 function special_case_ovs_upgrade_if_needed {
     66     if [[ -n $(rpm -q --scripts openvswitch | awk '/postuninstall/,/*/' | grep "systemctl.*try-restart") ]]; then
     67         echo "Manual upgrade of openvswitch - restart in postun detected" 
     68         rm -rf OVS_UPGRADE   <<============= 
     69         mkdir OVS_UPGRADE && pushd OVS_UPGRADE
     70         echo "Attempting to downloading latest openvswitch with yumdownloader"
     71         yumdownloader --resolve openvswitch
     72         for pkg in $(ls -1 *.rpm);  do
     73             if rpm -U --test $pkg 2>&1 | grep "already installed" ; then
     74                 echo "Looks like newer version of $pkg is already installed, skipping"
     75             else
     76                 echo "Updating $pkg with nopostun option"
     77                 rpm -U --replacepkgs --nopostun $pkg
     78             fi
     79         done
     80         popd
     81     else
     82         echo "Skipping manual upgrade of openvswitch - no restart in postun detected"
     83     fi
     84 
     85 }

~~~

In OSP 10 also we run the cleanup :
~~~
301 # Special-case OVS for https://bugs.launchpad.net/tripleo/+bug/1635205
302 # Update condition and add --notriggerun for +bug/1669714
303 function special_case_ovs_upgrade_if_needed {
304     # Always ensure yum has full cache
305     yum makecache || echo "Yum makecache failed. This can cause failure later on."
306     if rpm -qa | grep "^openvswitch-2.5.0-14" || rpm -q --scripts openvswitch | awk '/postuninstall/,/*/' | grep "systemctl.*try-restart" ; then
307         echo "Manual upgrade of openvswitch - ovs-2.5.0-14 or restart in postun detected"
308         rm -rf OVS_UPGRADE <<=========
309         mkdir OVS_UPGRADE && pushd OVS_UPGRADE
310         echo "Attempting to downloading latest openvswitch with yumdownloader"
311         yumdownloader --resolve openvswitch
312         for pkg in $(ls -1 *.rpm);  do
313             if rpm -U --test $pkg 2>&1 | grep "already installed" ; then
314                 echo "Looks like newer version of $pkg is already installed, skipping"
315             else
316                 echo "Updating $pkg with --nopostun --notriggerun"
317                 rpm -U --replacepkgs --nopostun --notriggerun $pkg
318             fi
319         done
320         popd
321     else
322         echo "Skipping manual upgrade of openvswitch - no restart in postun detected"
323     fi
324 
325 }

~~~

Comment 3 Marios Andreou 2017-11-20 10:33:16 UTC
Thanks Shatadru Bandyopadhyay - comment #2 is right .. we landed something to make the manual ovs upgrade more robust and cherrypicked to OSP9 but not OSP8 it seems. I'm going to post it and we can use this BZ for tracking that

Comment 4 Marios Andreou 2017-11-20 10:38:13 UTC
As per comment #2 this is in OSP9 with https://code.engineering.redhat.com/gerrit/#/c/112974/ ... so if this is upgrade of OSP8 to OSP9 they should already have this ^^ in their openstack-tripleo-heat-templates? 

I'm going to post the OSP8 cherrypick anyway, but if they upgrade their undercloud to OSP9 then they should have latest OSP9 tht with the more robust ovs upgrade from /#/c/112974/

Comment 22 Lon Hohberger 2018-10-04 10:36:13 UTC
According to our records, this should be resolved by openstack-tripleo-heat-templates-2.0.0-67.el7ost.  This build is available now.