Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1066445 - VMs are paused after migration from 3.2 to 3.3 hypervisor
Summary: VMs are paused after migration from 3.2 to 3.3 hypervisor
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.4.0
Assignee: Federico Simoncelli
QA Contact: Elad
URL:
Whiteboard: storage
Depends On:
Blocks: 1074298 rhev3.4beta 1142926
TreeView+ depends on / blocked
 
Reported: 2014-02-18 12:33 UTC by Tomas Dosek
Modified: 2018-12-05 17:19 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Virtual machines are no longer paused after migrations from 3.2 hypervisors to 3.3 hypervisors.
Clone Of:
: 1074298 (view as bug list)
Environment:
Last Closed: 2014-06-09 13:29:18 UTC
oVirt Team: Storage
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 727423 None None None Never
Red Hat Product Errata RHBA-2014:0504 normal SHIPPED_LIVE vdsm 3.4.0 bug fix and enhancement update 2014-06-09 17:21:35 UTC
Red Hat Knowledge Base (Solution) 727433 None None None Never

Description Tomas Dosek 2014-02-18 12:33:20 UTC
Description of problem:

This seems to be partially a result of fix in https://bugzilla.redhat.com/show_bug.cgi?id=1061621.

After we inspect and alter the XML file during migration, VM gets stuck on destination (newer RHEV-H  or in case of 3.3 -> 3.2 migration on the 3.2 hypervisor) with attribute error:

94:Feb 18 11:17:11 bl460-282 vdsm vm.Vm WARNING vmId=`44a60669-2931-4e5a-810d-43952d705644`::updating drive virtio-disk2 config path from /rhev/data-center/5a94dbe6-3fdc-4584-b434-318354a66c6f/7e43a8e9-0f61-4906-b2b8-cc4ef8a98429/images/1dc43ff9-4ff2-420b-9735-3fad4ecf6b20/49dd740e-6cfa-4c19-8a1b-e0acbc43e4d1 to /rhev/data-center/mnt/blockSD/7e43a8e9-0f61-4906-b2b8-cc4ef8a98429/images/1dc43ff9-4ff2-420b-9735-3fad4ecf6b20/49dd740e-6cfa-4c19-8a1b-e0acbc43e4d1
<-------- Path is getting updated during the migration
115:Feb 18 11:17:52 bl460-282 vdsm vm.Vm ERROR vmId=`44a60669-2931-4e5a-810d-43952d705644`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x24017d8>#012Traceback (most recent call last):#012  File "/usr/share/vdsm/sampling.py", line 351, in collect#012  File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  File "/usr/share/vdsm/vm.py", line 2313, in extendDrivesIfNeeded#012AttributeError: 'Drive' object has no attribute 'format'
116:Feb 18 11:17:54 bl460-282 vdsm vm.Vm ERROR vmId=`44a60669-2931-4e5a-810d-43952d705644`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x24017d8>#012Traceback (most recent call last):#012  File "/usr/share/vdsm/sampling.py", line 351, in collect#012  File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  File "/usr/share/vdsm/vm.py", line 2313, in extendDrivesIfNeeded#012AttributeError: 'Drive' object has no attribute 'format'
117:Feb 18 11:17:56 bl460-282 vdsm vm.Vm ERROR vmId=`44a60669-2931-4e5a-810d-43952d705644`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x24017d8>#012Traceback (most recent call last):#012  File "/usr/share/vdsm/sampling.py", line 351, in collect#012  File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  File "/usr/share/vdsm/vm.py", line 2313, in extendDrivesIfNeeded#012AttributeError: 'Drive' object has no attribute 'format'
118:Feb 18 11:17:58 bl460-282 vdsm vm.Vm ERROR vmId=`44a60669-2931-4e5a-810d-43952d705644`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x24017d8>#012Traceback (most recent call last):#012  File "/usr/share/vdsm/sampling.py", line 351, in collect#012  File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  File "/usr/share/vdsm/vm.py", line 2313, in extendDrivesIfNeeded#012AttributeError: 'Drive' object has no attribute 'format'
119:Feb 18 11:18:00 bl460-282 vdsm vm.Vm ERROR vmId=`44a60669-2931-4e5a-810d-43952d705644`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x24017d8>#012Traceback (most recent call last):#012  File "/usr/share/vdsm/sampling.py", line 351, in collect#012  File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  File "/usr/share/vdsm/vm.py", line 2313, in extendDrivesIfNeeded#012AttributeError: 'Drive' object has no attribute 'format'
120:Feb 18 11:18:02 bl460-282 vdsm vm.Vm ERROR vmId=`44a60669-2931-4e5a-810d-43952d705644`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x24017d8>#012Traceback (most recent call last):#012  File "/usr/share/vdsm/sampling.py", line 351, in collect#012  File "/usr/share/vdsm/sampling.py", line 226, in __call__#012  File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012  File "/usr/share/vdsm/vm.py", line 2313, in extendDrivesIfNeeded#012AttributeError: 'Drive' object has no attribute 'format'

Version-Release number of selected component (if applicable):
3.3.0-3

How reproducible:
vdsm-4.13.2-0.10

Steps to Reproduce:
1. have a 3.2 setup upgraded to 3.3
2. have a 3.2 hypervisor running VM and 3.3.0-3 hypervisor
3. migrate VM between the hypervisors

Actual results:
Argument error logged in the logs and VM gets paused.

Expected results:
Should work just fine

Comment 1 Tomas Dosek 2014-02-18 12:53:31 UTC
Actually it's the recovery files that are being rewritten indeed. But the result is the same. DST host constructs XML file with missing attributes for drives and hence VM pauses.

Comment 2 Dan Kenigsberg 2014-02-18 13:20:22 UTC
Could you share vdsm.log, too (from both source and destination hosts)?

Comment 9 Federico Simoncelli 2014-02-18 15:23:43 UTC
I have the feeling that what happened was:

1. the vm was created on vdsm 3.3.0 (< 3.3.0-3) with the new /rhev/data-center/mnt/... path

2. the vm was migrated to a 3.2 vdsm where it lost several attributes (e.g. format) because of the different path (the old /rhev/data-center/<spuuid>), this can be observed in bl460-282_vdsm.log (libvirtError: invalid argument: invalid path /rhev/data-center/mnt...)

3. the vm was migrated (or vdsm was upgraded in-place) to a vdsm 3.3.0-3 that thanks to the fix for bug 1061621 was able to recover the path (logs attached in the description) but not the rest (format)

Tomas can you confirm that this hypothesis is correct?

Comment 11 Tomas Dosek 2014-02-21 10:04:19 UTC
QE - note:

Please be sure all backward compatibility is tested next time. These particular scenarios were broken:

vm started on 3.2 => 3.3.0-3 OK
vm started on 3.2 => 3.3.0 NO!
vm started on 3.3.0 => 3.2 NO!
vm started on 3.3.0 => 3.3.0-3 OK (compatible mode)
vm started on 3.3.0-3 => 3.2 OK
vm started on 3.3.0-3 => 3.3.0 NO!

Comment 12 Tomas Dosek 2014-02-24 08:47:57 UTC
NO AIs for Engineering on this bug. The scenario was already fixed in 3.3.0-3.

Pushing to ON_QA - we need to make sure all the testing scenarios mentioned in comment 11 are covered ideally by automation.

After this the bug can be closed as NOTABUG or WONTFIX. We just need to cover this in QA process.

Resetting flags accordingly as well.

Comment 15 Elad 2014-03-12 15:13:09 UTC
The following combinations of vdsm versions were checked for VM migration. All passed.

4.13.2-0.5.el6ev  ---> vdsm-4.10.2-11.0.el6ev.x86_64     
vdsm-4.10.2-11.0.el6ev.x86_64  -->   4.13.2-0.5.el6ev   
vdsm-4.10.2-11.0.el6ev.x86_64  --> vdsm-4.12.0-105.git0da1561.el6ev.x86_64  
vdsm-4.12.0-105.git0da1561.el6ev.x86_64 --> vdsm-4.10.2-11.0.el6ev.x86_64
vdsm-4.10.2-11.0.el6ev.x86_64 --> vdsm-4.11.0-35.git34c1ef1.el6.x86_64



Moving to CLOSED WONTFIX

Comment 16 errata-xmlrpc 2014-06-09 13:29:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0504.html


Note You need to log in before you can comment on or make changes to this bug.