Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1690782 - [Upgrade] yum update stops OVN service during upgrade from 4.2 to 4.3
Summary: [Upgrade] yum update stops OVN service during upgrade from 4.2 to 4.3
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhv-openvswitch
Version: 2.3.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ovirt-4.3.3
: 4.3.0
Assignee: Miguel Duarte Barroso
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-20 08:58 UTC by Petr Matyáš
Modified: 2019-04-16 15:28 UTC (History)
7 users (show)

Fixed In Version: rhv-openvswitch-2.10-5.el7ev.noarch
Doc Type: Bug Fix
Doc Text:
This fix ensures that installing the OVS-2.10 package restarts the OVS/OVN services after the package completes the install and update process.
Clone Of:
Environment:
Last Closed:
oVirt Team: Network
Target Upstream Version:


Attachments (Terms of Use)
engine server and ovn logs (deleted)
2019-03-20 08:58 UTC, Petr Matyáš
no flags Details
logs from manual reproduction (deleted)
2019-03-20 09:09 UTC, Michael Burman
no flags Details
log of updating oVirt (deleted)
2019-03-20 11:00 UTC, Dominik Holler
no flags Details
log of updating rhvm (deleted)
2019-03-20 11:01 UTC, Dominik Holler
no flags Details

Description Petr Matyáš 2019-03-20 08:58:06 UTC
Created attachment 1545947 [details]
engine server and ovn logs

Description of problem:
After upgrade of engine from 4.2 to 4.3 and running a few tests, we restarted the engine and sync of ovn networks failed.

Version-Release number of selected component (if applicable):
ovirt-provider-ovn-1.2.20-1.el7ev.noarch

How reproducible:
always

Steps to Reproduce:
1. upgrade engine from 4.2 to 4.3
2. change default provider in cluster to ovn
3. will evoke ovn networks sync

Actual results:
2019-03-19 20:58:44,636+02 INFO  [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engi
neScheduled-Thread-47) [5fba6d90] Running command: SyncNetworkProviderCommand internal: true.
2019-03-19 20:58:44,728+02 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engi
neScheduled-Thread-47) [5fba6d90] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineExcept
ion: (Failed with error Connection refused (Connection refused) and code 5050)
2019-03-19 20:58:44,756+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engin
eScheduled-Thread-47) [5fba6d90] EVENT_ID: PROVIDER_SYNCHRONIZED_FAILED(216), Failed to synchronize networks of Provider ovirt-provide
r-ovn.

Expected results:
synced networks

Additional info:
after restart of ovn we have this in ovn log:
2019-03-20 08:59:06,714 root Incorrect path: /v2.0
Traceback (most recent call last):
  File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request
    method, path_parts, content
  File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 173, in handle_request
    self.get_responses(), method, path_parts
  File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 193, in get_response_handler
    raise PathNotFoundError()
PathNotFoundError

Comment 1 Michael Burman 2019-03-20 09:06:49 UTC
The real issue here is the ovirt-provider-ovn is killed during the upgrade from 4.2 >> 4.3.
This causing the sync to fail of course. What is really weird here is that it is caused the infra job to stuck for ever and it can't finished(only can abort).

systemd: Stopped OVN northd management daemon. 
after that, nothing and no one bringing the service up.

Active: inactive (dead) since Wed 2019-03-20 10:48:36 IST; 16min ago
  Main PID: 9416 (code=killed, signal=TERM)

Managed to reproduce the issue manually.

Comment 2 Michael Burman 2019-03-20 09:09:11 UTC
Created attachment 1545948 [details]
logs from manual reproduction

Comment 3 Dominik Holler 2019-03-20 11:00:55 UTC
Created attachment 1546009 [details]
log of updating oVirt

upstream seems to be not affected

Comment 4 Dominik Holler 2019-03-20 11:01:49 UTC
Created attachment 1546012 [details]
log of updating rhvm

"yum update" on downstream seems to trigger this bug

Comment 5 Dan Kenigsberg 2019-03-20 13:43:30 UTC
We can add a %pre script checking the status of OvS services, storing it in /var/run/openvswitch/somrthing, and then re-applying the state during %post.

Comment 7 Dan Kenigsberg 2019-03-20 14:52:24 UTC
I have not tried, but I hope that the following would work around our problem.

diff --git a/rhv-openvswitch.spec b/rhv-openvswitch.spec
index 5c87ce8..f77ba09 100644
--- a/rhv-openvswitch.spec
+++ b/rhv-openvswitch.spec
@@ -95,6 +95,26 @@ Wrapper rpm for the base openvswitch-ovn-common package
 %files ovn-vtep
 %files ovn-common
 
+%pre
+# record which service were enabled before upgrade
+preenabled_dir=/var/run/rhv-openvswitch/enabled
+mkdir -p "$preenabled_dir"
+for service in openvswitch ovn-northd; do
+    if [ `systemctl is-enabled "$service"` = enabled ]; then
+        touch "$preenabled_dir/$service"
+    fi
+done
+
+%post
+preenabled_dir=/var/run/rhv-openvswitch/enabled
+if [ -d "$preenabled_dir" ]; then
+    for service in openvswitch ovn-northd; do
+        if [ -e "$preenabled_dir/$service" ]; then
+            systemctl enable --now "$service"
+            rm "$preenabled_dir/$service"
+        fi
+    done
+fi
 
 %changelog
 * Mon Oct 15 2018 Miguel Duarte Barroso <mdbarroso@redhat.com> - 1:2.10-3

Comment 14 Michael Burman 2019-03-31 06:02:11 UTC
Verified on - rhv-openvswitch.noarch 1:2.10-5.el7ev
ovirt-engine-setup.noarch 0:4.3.3.1-0.1.el7
rhv-openvswitch-2.10-5.el7ev.noarch

yum update from ovirt-engine-setup-4.2.8.5-0.1.el7ev.noarch >> 4.3.3.1-0.1.el7
after the update, ovirt-provider-ovn is kept alive.

ovirt-provider-ovn.service - oVirt OVN provider
   Loaded: loaded (/usr/lib/systemd/system/ovirt-provider-ovn.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-03-31 08:27:09 IDT; 2min 8s ago
 Main PID: 26573 (python)
   CGroup: /system.slice/ovirt-provider-ovn.service
           └─26573 /usr/bin/python /usr/share/ovirt-provider-ovn/ovirt_provider_ovn.py


Note You need to log in before you can comment on or make changes to this bug.