Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 591609

Summary: Migrating guests in RHEVM generates exception traceback emails from Satellite
Product: Red Hat Satellite 5 Reporter: Martin Osvald 🛹 <mosvald>
Component: VirtualizationAssignee: Milan Zázrivec <mzazrivec>
Status: CLOSED ERRATA QA Contact: Dimitar Yordanov <dyordano>
Severity: medium Docs Contact:
Priority: medium    
Version: 530CC: apevec, brian, cperry, dyordano, hchiramm, jhutar, rjones
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, after migrating a Xen guest domain to KVM using Red Hat Enterprise Virtualization Manager, the following error may have occurred: While running 'registration.virt_notify': caught exceptions.KeyError : 'Unknown' This was caused by the use of invalid machine state, "Unknown". With this update, the incorrect state has been changed to "nostate".
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-11 14:34:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 518253    
Attachments:
Description Flags
email with exception none

Description Martin Osvald 🛹 2010-05-12 17:12:40 UTC
Created attachment 413493 [details]
email with exception

Description of problem:

If vdsmd doesn't return state of the machine in domain info, poller.py sends "Unknown" machine state to Satellite which causes exception, because Satellite doesn't know about "Unknown" state.


Version-Release number of selected component (if applicable):

Satellite 5.3 - spacewalk-backend-0.5.28-54
rhn-virtualization-host-5.3.0-17


How reproducible:

always


Steps to Reproduce:

1. Migrate xen to kvm guest through RHEVM (vdsmd has to be enabled at host)

  
Actual results:

see attached email (sensitive data removed)


Expected results:

no exception emails should be sent by Satellite


Additional info:

related client side source code (related code is concatenated with '<<<---'):

===< snip from client/tools/rhn-virtualization/virtualization/poller.py ===
def poll_through_vdsm():
   """
    This method polls all the virt guests running on a VDSM enabled Host.
    Libvirt is disabled by default on RHEV-M managed clients.
    * Imports the localvdsm client that talks to the localhost
      and fetches the list of vms and their info.
    * Extract the data and construct the state to pass it to the 
      execution plan for guest polling.
    * The server should account for business rules similar to
      xen/kvm.
   """
   import localvdsm
   try:
       server = localvdsm.connect()
   except:
       # VDSM raised an exception we're done here
       return {}
   # Extract list of vm's. True returns full list
   try:
       domains = server.list(True)
   except:
       # Something went wrong in vdsm, exit
       return {}

   if not len(domains['vmList']):
       # No domains, exit.
       return

   state = {}
   for domain in domains['vmList']:
       #trim uuid
       uuid = domain['vmId'].lower().replace('-', '')
       # Map the VDSM status to libvirt for server compatibility
       status = "Unknown"          <<<---
       if VIRT_VDSM_STATUS_MAP.has_key(domain['status']):  <<<---
           status = VIRT_VDSM_STATUS_MAP[domain['status']]
       # This is gonna be fully virt as its managed by VDSM
       virt_type = VirtualizationType.FULLY

       #Memory
       memory = int(domain['memSize']) * 1024

       # vcpus
       if domain.has_key('smp'):
           vcpus = domain['smp']
       else:
           vcpus = '1'

       properties = {
           PropertyType.NAME   : domain['vmName'],
           PropertyType.UUID   : uuid,
           PropertyType.TYPE   : virt_type,
           PropertyType.MEMORY : memory, # current memory
           PropertyType.VCPUS  : vcpus,
           PropertyType.STATE  : status}

       state[uuid] = properties

   if state: _log_debug("Polled state: %s" % repr(state))

   return state
=== snip >===

There is also probably a bug in routine which migrate host from xen to kvm with not updating domain status correctly through vdsm during migration, but the fact that the above code sends/could send unknown state for Satellite is a bug. It shouldn't send such state or should send one of the known states:

NOSTATE     = 'nostate'
RUNNING     = 'running'
BLOCKED     = 'blocked'
PAUSED      = 'paused'
SHUTDOWN    = 'shutdown'
SHUTOFF     = 'shutoff'
CRASHED     = 'crashed'

or there should be implemented another state named "Unknown".

Traceback from Satellite side (see attached email for complete info):

Exception Handler Information
Traceback (most recent call last):
  File "/usr/share/rhn/server/apacheRequest.py", line 120, in call_function
    response = apply(func, params)
  File "/usr/share/rhn/server/handlers/xmlrpc/registration.py", line 957, in virt_notify
    rhnVirtualization._virt_notify(server_id, actions)
  File "/usr/share/rhn/server/rhnVirtualization.py", line 844, in _virt_notify
    handler.handle(server_id, action)
  File "/usr/share/rhn/server/rhnVirtualization.py", line 214, in handle
    self.__convert_properties(properties)
  File "/usr/share/rhn/server/rhnVirtualization.py", line 773, in __convert_properties
    properties[PropertyType.STATE] = CLIENT_SERVER_STATE_MAP[state]
KeyError: 'Unknown'

Comment 3 Matthew Booth 2010-06-03 10:10:21 UTC
There's a vocab nit which becomes important in this kind of discussion: we talk about 'converting' a guest from Xen to KVM, and 'migrating' a guest from hypervisor node A to hypervisor node B.

If I'm reading this correctly, a Xen guest has been converted to run on RHEV. It subsequently can't be migrated. Is this correct? How was it converted?

Comment 5 Alan Pevec 2010-06-07 15:55:26 UTC
Do you have a VM in this state? If yes, please run on the Hypervisor: vdsClient -s 0 list

Comment 8 Milan Zázrivec 2010-07-15 16:18:53 UTC
spacewalk.git master:

commit a4be5dcb417d1ad218c205f5cae6adc278f76bf6
Author: Milan Zazrivec <mzazrivec@redhat.com>
Date:   Thu Jul 15 18:14:35 2010 +0200

    591609 - 'Unknown' is not a valid virt. guest state
    
    'nostate' is the valid fallback state value

Comment 9 Milan Zázrivec 2010-07-15 16:21:57 UTC
satellite.git SATELLITE-5.3: 192a1ff8f74dab587d3fc0d21cd8b363d6dbe9c3

Comment 12 Dimitar Yordanov 2010-09-01 08:55:06 UTC
Hi Martin,

 I have just checked that the patch is included in  package rhn-virtualization-5.3.0-18.el5sat.src.rpm

+- 591609 - 'Unknown' is not a valid virt. guest state
-        status = "Unknown"
+        status = 'nostate'

 Unfortunately I faced a problem. I have no access to RHEV-M.
 I would like to ask you to if you can install  rhn-virtualization-common-5.3.0-18.el5sat.noarch.rpm  and  rhn-virtualization-common-5.3.0-18.el5sat.noarch.rpm  on the target machine and let us know if the issue is fixed correctly.
 Thank you in advance.

Regards
Dimitar Yordanov

Comment 13 Jaromir Hradilek 2010-09-03 15:53:15 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, after migrating a Xen guest domain to KVM using Red Hat Enterprise Virtualization Manager (RHEV-M), the following error may have occurred:

  While running 'registration.virt_notify': caught exceptions.KeyError : 'Unknown'

This was caused by the use of invalid machine state, "Unknown". With this update, the incorrect state has been changed to "nostate".

Comment 14 Jaromir Hradilek 2010-09-06 08:20:26 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,4 +1,4 @@
-Previously, after migrating a Xen guest domain to KVM using Red Hat Enterprise Virtualization Manager (RHEV-M), the following error may have occurred:
+Previously, after migrating a Xen guest domain to KVM using Red Hat Enterprise Virtualization Manager, the following error may have occurred:
 
   While running 'registration.virt_notify': caught exceptions.KeyError : 'Unknown'

Comment 21 errata-xmlrpc 2010-10-11 14:34:50 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0761.html