Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1369415 - [z-stream clone - 3.6.9] [InClusterUpgrade] Possible race condition with large amount of VMs in cluster
Summary: [z-stream clone - 3.6.9] [InClusterUpgrade] Possible race condition with larg...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.6.8
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: ovirt-3.6.9
: ---
Assignee: Arik
QA Contact: sefi litmanovich
URL:
Whiteboard:
Depends On: 1366786
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-23 11:09 UTC by rhev-integ
Modified: 2017-04-03 13:08 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the update of the compatibility version of a cluster with many running virtual machines that are installed with the guest-agent caused a deadlock that caused the update to fail. In some cases, these clusters could not be upgraded to a newer compatibility version. Now, the deadlock in the database has been prevented so that a cluster with many running virtual machines that are installed with the guest-agent can be upgraded to newer compatibility version.
Clone Of: 1366786
Environment:
Last Closed: 2016-09-21 18:06:00 UTC
oVirt Team: Virt
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 62372 ovirt-engine-4.0 MERGED core: fix monitoring of guest agent nics 2016-08-23 11:09:35 UTC
oVirt gerrit 62392 ovirt-engine-3.6 MERGED core: Cluster update updates VMs only if necessary 2016-08-23 11:09:35 UTC
oVirt gerrit 62514 ovirt-engine-4.0 MERGED core: make VmBase comparable 2016-08-23 11:09:35 UTC
oVirt gerrit 62515 ovirt-engine-4.0 MERGED core: update cluster to query only static vm data 2016-08-23 11:09:35 UTC
oVirt gerrit 62516 ovirt-engine-4.0 MERGED core: determine the order of vm statistics updates 2016-08-23 11:09:35 UTC
oVirt gerrit 62517 ovirt-engine-4.0 MERGED core: determine the order of guest agent nic updates 2016-08-23 11:09:35 UTC
oVirt gerrit 62518 ovirt-engine-4.0 MERGED core: fix possible deadlock on update cluster version 2016-08-23 11:09:35 UTC
oVirt gerrit 62521 master MERGED core: make VmBase comparable 2016-08-23 11:09:35 UTC
oVirt gerrit 62522 master MERGED core: update cluster to query only static vm data 2016-08-23 11:09:35 UTC
oVirt gerrit 62523 master MERGED core: determine the order of vm statistics updates 2016-08-23 11:09:35 UTC
oVirt gerrit 62524 master MERGED core: determine the order of guest agent nic updates 2016-08-23 11:09:35 UTC
oVirt gerrit 62525 master MERGED core: fix possible deadlock on update cluster version 2016-08-23 11:09:35 UTC
Red Hat Knowledge Base (Solution) 2525531 None None None 2016-08-23 11:09:35 UTC
Red Hat Product Errata RHSA-2016:1929 normal SHIPPED_LIVE Moderate: Red Hat Virtualization Manager (RHV) bug fix 3.6.9 2016-09-21 21:57:10 UTC
oVirt gerrit 62637 ovirt-engine-3.6 MERGED core: fix monitoring of guest agent nics 2016-08-23 11:09:35 UTC

Comment 2 sefi litmanovich 2016-08-29 17:02:24 UTC
Verified with rhevm-3.6.9-0.1.el6.noarch.

Had a cluster with 126 Vms running.
Changed cluster compatibility version from 3.5 to 3.6 (hosts were 3.6 all the time) and monitored the updateVm calls with tail on engine log.
Repeated this several times (each time setting the cluster compatibility back to 3.5 via DB.
Ran the upgrade for 5 times, no race has occurred.
Please advise if this test isn't sufficient.

Comment 3 sefi litmanovich 2016-08-30 09:04:47 UTC
I see this test was approved in the 4.0.4 version of it. 10x.

Comment 5 errata-xmlrpc 2016-09-21 18:06:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-1929.html

Comment 6 Frank DeLorey 2016-10-26 20:12:50 UTC
I have a customer running 3.6.9 that just hit this in a cluster that only has 30 VMs. I am gathering all the data and will open a new BZ.

Regards,

Frank

Comment 8 Frank DeLorey 2016-10-27 10:06:40 UTC
This is happening on every attempt to upgrade the cluster. I will grab the engine log and post the related errors into this BZ.

Frank

Comment 9 Michal Skrivanek 2016-10-29 06:14:34 UTC
(In reply to Frank DeLorey from comment #8)
> This is happening on every attempt to upgrade the cluster. I will grab the
> engine log and post the related errors into this BZ.
> 
> Frank

It is a different issue, tracked in bug 1389764 now


Note You need to log in before you can comment on or make changes to this bug.