Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1059482 - Migrating between older and newer RHEV-H images leads to flood "path /rhev/data-center/mnt/blockSD/<UUID>/images/<UUID>/<UUID> not assigned to domain"
Summary: Migrating between older and newer RHEV-H images leads to flood "path /rhev/da...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.4.0
Assignee: Federico Simoncelli
QA Contact: Leonid Natapov
URL:
Whiteboard: storage
: 1055437 (view as bug list)
Depends On:
Blocks: rhev3.4beta 1094944 1142926
TreeView+ depends on / blocked
 
Reported: 2014-01-29 23:07 UTC by David Gibson
Modified: 2018-12-09 17:28 UTC (History)
29 users (show)

Fixed In Version: ovirt-3.4.0-beta3
Doc Type: Bug Fix
Doc Text:
An update to VDSM changed a path used by libvirt. Consequently, logs were flooded with messages about invalid paths. The virtual machines associated with these logs worked, but migrations from older RHEV-H to newer RHEV-H caused the logs to flood. An update to VDSM adjusts the libvirt XML to match the form the destination is using after a migration from an older version of RHEV-H to a newer version of RHEV-H. Now, logs are no longer flooded with messages about invalid paths.
Clone Of:
Environment:
Last Closed: 2014-06-09 13:28:32 UTC
oVirt Team: Storage
Target Upstream Version:
scohen: needinfo+
scohen: needinfo+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 24202 None None None Never
oVirt gerrit 24315 master MERGED sd: fix volume path returned by linkBCImage Never
oVirt gerrit 24324 None None None Never
oVirt gerrit 29485 ovirt-3.5 MERGED sd: fix volume path returned by linkBCImage Never
Red Hat Knowledge Base (Solution) 701473 None None None Never
Red Hat Product Errata RHBA-2014:0504 normal SHIPPED_LIVE vdsm 3.4.0 bug fix and enhancement update 2014-06-09 17:21:35 UTC

Description David Gibson 2014-01-29 23:07:17 UTC
Description of problem:

Customer found that after updating RHEV-H and migrating VMs to the updated host, logs were flooded with messages like:

Jan 29 05:17:39 rhevh04 vdsm vm.Vm ERROR vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>#012Traceback (most recent call last):#012  File "/usr/share/vdsm/sampling.py", line 351, in collect#012  File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded#012  File "/usr/share/vdsm/vm.py", line 842, in f#012  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper#012  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo#012libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain

The VMs otherwise work.

The problem is because the newer vdsm version configures libvirt to use paths of the form:

/rhev/data-center/mnt/<mountpoint>/<sd uuid>/images/<disk uuid>/<img uuid>
                  ^^^^^^^^^^^^^^^^

whereas the older vdsm used the form

/rhev/data-center/<sp uuid>/<sd uuid>/images/<disk uuid>/<img uuid>
                  ^^^^^^^^^

For VMs started under the new vdsm this is fine, but for VMs migrated from an older hypervisor, the libvirt XML, including the old-style paths is carried along.  That means that the vdsm and libvirt paths are out of sync, causing the error above when vdsm attempts to query libvirt for information about the disk.

Version-Release number of selected component (if applicable):

vdsm 4.13.2-0.6 uses the new path format

How reproducible:

100%

Steps to Reproduce:
1. Have a RHEV setup with pre-vdsm-4.13.2 hypervisors
2. Update one hypervisor to the RHEL 6.5 based image with vdsm-4.13.2
3. Migrate a VM from the old to the new hypervisor

Actual results:

VM works, but logs are spammed with libvirt and vdsm errors.

Expected results:

VM works, without extraneous errors.

Additional info:

This can be worked around by restarting the VMs under the new hypervisor, but obviously customers prefer not to do that.

Comment 3 David Gibson 2014-02-03 04:06:44 UTC
In fact, fixing only for 3.4.0 could actually be *worse* than not fixing it at all.  Because the problem is a mismatch in behaviour between hypervisor versions, just reverting the behaviour in 3.4 would cause this problem to occur again for all those people who've switched to the current behaviour in the interim, and restarted their VMs to work around the errors.

A real fix would need to check for this situation during migrate, and adjust the libvirt XML to match whatever form the destination vdsm is using.

Comment 10 David Gibson 2014-02-11 23:08:31 UTC
@Ulhas,

Which workaround from the KCS?  Restarting the VM, or migrating back to the old vdsm?

Comment 11 Ulhas Surse 2014-02-12 04:26:59 UTC
David, 
It's restart the VM.

2) Otherwise if the VM can be shutdown then shut it down on the vdsm 4.13.2-0.6 hypervisor and start it again. Note it must be shutdown completely and not simply rebooted.

Comment 18 Nir Soffer 2014-02-26 19:50:59 UTC
*** Bug 1055437 has been marked as a duplicate of this bug. ***

Comment 19 Leonid Natapov 2014-02-27 06:51:34 UTC
Tested with two RHEVH nodes and 3.4 management.
First RHEVH - RHEL 6.4 with vdsm-4.10.2-25.1.el6ev
Second RHEVH - RHEL 6.5 with vdsm-4.13.2-0.6.el6ev.

VM migrated from the first rhevh to the second one and vise versa.
No error appear in vdsm.log.

Comment 20 errata-xmlrpc 2014-06-09 13:28:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0504.html


Note You need to log in before you can comment on or make changes to this bug.