Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1687346 - VM migration fails with "Attempt to migrate guest to the same host" error
Summary: VM migration fails with "Attempt to migrate guest to the same host" error
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rhhi
Version: rhhiv-1.6
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Sahina Bose
QA Contact: bipin
URL:
Whiteboard:
Depends On: 1687340
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-11 10:44 UTC by bipin
Modified: 2019-03-27 07:50 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1687340
Environment:
Last Closed: 2019-03-27 07:50:25 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description bipin 2019-03-11 10:44:45 UTC
+++ This bug was initially created as a clone of Bug #1687340 +++

Description of problem:
=======================
Migration of vm's error out suggesting that it cant be migrated to same host. This is seen on rhv 4.3.  


Version-Release number of selected component
============================================
rhvh-4.3.0.5-0.20190305.0+1
glusterfs-3.12.2-46
kernel 3.10.0-957.10.1.el7.x86_64
vdsm-4.30.10-1.el7ev.x86_64
libvirt-4.5.0-10.el7_6.6.x86_64
ovirt-engine-4.3.2-0.1.el7.noarch

How reproducible:
================
1/1


Steps to Reproduce:
==================
1.Setup RHHI 1.6 enviornment
2.Create vm's using the RHV-M. (Compute --> Virtual machines --> New)
3.Once the vm's are up, try migrating 

Actual results:
==============
The migration fails

Expected results:
================
Migration shouldn't fail

Additional info:
===============

--- Additional comment from bipin on 2019-03-11 10:34:29 UTC ---

Engine log:
==========
2019-03-11 14:59:02,546+05 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-8) [] Migration of VM 'vm1' to host 'rhsqa-grafton9-nic2.lab.eng.blr.redhat.com' failed: VM destroyed during the startup.
2019-03-11 14:59:02,547+05 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [] VM '5fc476c6-7c3b-4635-ae7c-d6f0d95b146b'(vm1) moved from 'MigratingFrom' --> 'Up'
2019-03-11 14:59:02,547+05 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [] Adding VM '5fc476c6-7c3b-4635-ae7c-d6f0d95b146b'(vm1) to re-run list
2019-03-11 14:59:02,604+05 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-15) [] Rerun VM '5fc476c6-7c3b-4635-ae7c-d6f0d95b146b'. Called from VDS 'rhsqa-grafton7-nic2.lab.eng.blr.redhat.com'
2019-03-11 14:59:02,608+05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-4941) [] START, MigrateStatusVDSCommand(HostName = rhsqa-grafton7-nic2.lab.eng.blr.redhat.com, MigrateStatusVDSCommandParameters:{hostId='386144e8-1bc7-4f20-8973-b12f142b0a39', vmId='5fc476c6-7c3b-4635-ae7c-d6f0d95b146b'}), log id: 94a610f
2019-03-11 14:59:02,611+05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-4941) [] FINISH, MigrateStatusVDSCommand, return: , log id: 94a610f
2019-03-11 14:59:02,731+05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-4941) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed  (VM: vm1, Source: rhsqa-grafton7-nic2.lab.eng.blr.redhat.com, Destination: rhsqa-grafton9-nic2.lab.eng.blr.redhat.com)


VDSM:
====
2019-03-11 14:59:01,076+0530 INFO  (migsrc/5fc476c6) [virt.vm] (vmId='5fc476c6-7c3b-4635-ae7c-d6f0d95b146b') Migration semaphore: acquiring (migration:407)
2019-03-11 14:59:01,076+0530 INFO  (migsrc/5fc476c6) [virt.vm] (vmId='5fc476c6-7c3b-4635-ae7c-d6f0d95b146b') Migration semaphore: acquired (migration:409)
2019-03-11 14:59:01,955+0530 INFO  (migsrc/5fc476c6) [virt.vm] (vmId='5fc476c6-7c3b-4635-ae7c-d6f0d95b146b') Creation of destination VM took: 0 seconds (migration:469)
2019-03-11 14:59:01,955+0530 INFO  (migsrc/5fc476c6) [virt.vm] (vmId='5fc476c6-7c3b-4635-ae7c-d6f0d95b146b') starting migration to qemu+tls://rhsqa-grafton9-nic2.lab.eng.blr.redhat.com/system with miguri tcp://10.70.45.31 (migration:498)
2019-03-11 14:59:02,205+0530 ERROR (migsrc/5fc476c6) [virt.vm] (vmId='5fc476c6-7c3b-4635-ae7c-d6f0d95b146b') internal error: Attempt to migrate guest to the same host c51f28d8-cd98-4a0d-9f1f-8da7e996106f (migration:282)
2019-03-11 14:59:02,548+0530 ERROR (migsrc/5fc476c6) [virt.vm] (vmId='5fc476c6-7c3b-4635-ae7c-d6f0d95b146b') Failed to migrate (migration:450)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run
    time.time(), migrationParams, machineParams
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration
    self._perform_with_conv_schedule(duri, muri)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule
    self._perform_migration(duri, muri)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration
    self._migration_flags)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1779, in migrateToURI3
    if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
libvirtError: internal error: Attempt to migrate guest to the same host c51f28d8-cd98-4a0d-9f1f-8da7e996106f
2019-03-11 14:59:02,611+0530 INFO  (jsonrpc/4) [api.virt] START getMigrationStatus() from=::ffff:10.70.36.247,58600, vmId=5fc476c6-7c3b-4635-ae7c-d6f0d95b146b (api:48)
2019-03-11 14:59:02,611+0530 INFO  (jsonrpc/4) [api.virt] FINISH getMigrationStatus return={'status': {'message': 'Done', 'code': 0}, 'migrationStats': {'status': {'message': 'Fatal error during migration', 'code': 12}, 'progress': 0}} from=::ffff:10.70.36.247,58600, vmId=5fc476c6-7c3b-4635-ae7c-d6f0d95b146b (api:54)

Libvirt debug logs:
==================
2019-03-11 09:29:01.994+0000: 27741: debug : qemuDomainDefFormatBufInternal:7318 : Removing default USB controller from domain 'vm1' for migration compatibility
2019-03-11 09:29:01.994+0000: 27741: debug : qemuDomainDefFormatBufInternal:7341 : Removing default pci-root from domain 'vm1' for migration compatibility
2019-03-11 09:29:01.994+0000: 27741: info : virObjectRef:382 : OBJECT_REF: obj=0x7f2570149660
2019-03-11 09:29:01.994+0000: 27741: debug : virStringMatch:1189 : match '/var/lib/libvirt/qemu/channels/5fc476c6-7c3b-4635-ae7c-d6f0d95b146b.ovirt-guest-agent.0' for '^/var/lib/libvirt/qemu/channel/target/([^/]
+\.)|(domain-[^/]+/)ovirt-guest-agent\.0$'
2019-03-11 09:29:01.994+0000: 27741: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f2570149660
2019-03-11 09:29:01.994+0000: 27741: info : virObjectRef:382 : OBJECT_REF: obj=0x7f2570149660
2019-03-11 09:29:01.994+0000: 27741: debug : virStringMatch:1189 : match '/var/lib/libvirt/qemu/channels/5fc476c6-7c3b-4635-ae7c-d6f0d95b146b.org.qemu.guest_agent.0' for '^/var/lib/libvirt/qemu/channel/target/([
^/]+\.)|(domain-[^/]+/)org\.qemu\.guest_agent\.0$'
2019-03-11 09:29:01.995+0000: 27741: debug : qemuMigrationSrcPerformPeer2Peer3:4121 : Prepare3 0x7f25d8003df0
2019-03-11 09:29:01.995+0000: 27741: debug : qemuDomainObjEnterRemote:7182 : Entering remote (vm=0x7f25702b4530 name=vm1)
2019-03-11 09:29:02.203+0000: 27741: error : virNetClientProgramDispatchError:174 : internal error: Attempt to migrate guest to the same host c51f28d8-cd98-4a0d-9f1f-8da7e996106f
2019-03-11 09:29:02.203+0000: 27741: debug : qemuDomainObjExitRemote:7193 : Exited remote (vm=0x7f25702b4530 name=vm1)
2019-03-11 09:29:02.204+0000: 27741: debug : qemuDomainObjEnterRemote:7182 : Entering remote (vm=0x7f25702b4530 name=vm1)
2019-03-11 09:29:02.204+0000: 27741: debug : virConnectUnregisterCloseCallback:1410 : conn=0x7f25d8003df0
2019-03-11 09:29:02.204+0000: 27741: debug : qemuDomainObjExitRemote:7193 : Exited remote (vm=0x7f25702b4530 name=vm1)
2019-03-11 09:29:02.204+0000: 27741: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f2570149660
2019-03-11 09:29:02.204+0000: 27740: debug : virFileClose:111 : Closed fd 57
2019-03-11 09:29:02.204+0000: 27741: debug : qemuMigrationSrcRestoreDomainState:137 : driver=0x7f2570125af0, vm=0x7f25702b4530, pre-mig-state=running, state=running, reason=booted
2019-03-11 09:29:02.204+0000: 27740: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25d8006c60
2019-03-11 09:29:02.204+0000: 27740: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7f25d8006c60
2019-03-11 09:29:02.204+0000: 27741: debug : qemuDomainObjEndAsyncJob:6998 : Stopping async job: migration out (vm=0x7f25702b4530 name=vm1)
2019-03-11 09:29:02.204+0000: 27740: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25d80244e0
2019-03-11 09:29:02.204+0000: 27740: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7f25d80244e0
2019-03-11 09:29:02.204+0000: 27740: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25d80234a0
2019-03-11 09:29:02.204+0000: 27741: info : virObjectRef:382 : OBJECT_REF: obj=0x7f2570149660
2019-03-11 09:29:02.204+0000: 27740: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7f25d80234a0
2019-03-11 09:29:02.205+0000: 27740: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25d8006bf0
2019-03-11 09:29:02.205+0000: 27740: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7f25d8006bf0
2019-03-11 09:29:02.205+0000: 27740: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25d80243c0
2019-03-11 09:29:02.205+0000: 27740: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7f25d80243c0
2019-03-11 09:29:02.205+0000: 27740: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25d80068e0
2019-03-11 09:29:02.205+0000: 27740: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7f25d80068e0
2019-03-11 09:29:02.205+0000: 27740: debug : virFileClose:111 : Closed fd 59
2019-03-11 09:29:02.205+0000: 27740: debug : virFileClose:111 : Closed fd 58
2019-03-11 09:29:02.205+0000: 27741: debug : virFileMakePathHelper:3093 : path=/var/run/libvirt/qemu mode=0777
2019-03-11 09:29:02.205+0000: 27741: debug : virFileClose:111 : Closed fd 57
2019-03-11 09:29:02.205+0000: 27741: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f2570149660
2019-03-11 09:29:02.205+0000: 27741: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25702b4530
2019-03-11 09:29:02.205+0000: 27741: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f2570149660
2019-03-11 09:29:02.205+0000: 27741: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25d8004640
2019-03-11 09:29:02.205+0000: 27741: info : virObjectUnref:346 : OBJECT_DISPOSE: obj=0x7f25d8004640
2019-03-11 09:29:02.205+0000: 27741: debug : virDomainDispose:318 : release domain 0x7f25d8004640 vm1 5fc476c6-7c3b-4635-ae7c-d6f0d95b146b
2019-03-11 09:29:02.205+0000: 27741: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25d8001440
2019-03-11 09:29:02.205+0000: 27741: debug : virThreadJobClear:121 : Thread 27741 (virNetServerHandleJob) finished job remoteDispatchDomainMigratePerform3Params with ret=-1
2019-03-11 09:29:02.205+0000: 27741: info : virObjectUnref:344 : OBJECT_UNREF: obj=0x7f25b8004260


Note You need to log in before you can comment on or make changes to this bug.