Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1690862 - The engine fails to migrate the hosted-engine VM with "Could not fetch data needed for VM migrate operation"
Summary: The engine fails to migrate the hosted-engine VM with "Could not fetch data n...
Keywords:
Status: CLOSED DUPLICATE of bug 1670701
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Virt
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: ---
: ---
Assignee: Michal Skrivanek
QA Contact: meital avital
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-20 11:11 UTC by Simone Tiraboschi
Modified: 2019-03-20 11:20 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-20 11:19:11 UTC
oVirt Team: Virt


Attachments (Terms of Use)
he_migrate.png (deleted)
2019-03-20 11:13 UTC, Simone Tiraboschi
no flags Details
engine.log (deleted)
2019-03-20 11:14 UTC, Simone Tiraboschi
no flags Details

Description Simone Tiraboschi 2019-03-20 11:11:44 UTC
Description of problem:
We have an environment with an hosted-engine cluster with 2 hosts.

The two hosts are correctly deployed and configured for hosted-engine, both have an HA score of 3400 points and are up and running for the engine.

At cluster level we have:
Intel SandyBridge IBRS SSBD Family

At guest level:
Guest CPU Type: Intel SandyBridge IBRS SSBD Family

and on both the hosts:
CPU Type:
Intel SandyBridge IBRS SSBD Family

Kernel Features:
PTI: 1, IBRS: 0, RETP: 1

when we try to migrate the hosted-engine VM from the engine we get a warning with  "Could not fetch data needed for VM migrate operation" (see the attached screenshot).

in engine.log we see just:
2019-03-20 10:44:14,789Z INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-51) [d03b77ca-4c67-4303-b27e-e41d828fb174] Candidate host 'host_mixed_1' ('42a138b6-efd6-4a7e-a6b2-d962b279f9ec') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU' (correlation id: null)
2019-03-20 10:44:14,789Z INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-51) [d03b77ca-4c67-4303-b27e-e41d828fb174] Candidate host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' ('c83a8008-cd71-4671-9f10-1325f3364034') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU' (correlation id: null)


it's worth to mention that in the past we got:
2019-03-20 09:45:51,290Z ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com', last response arrived 16876 ms ago.
2019-03-20 09:48:31,204Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-8) [] EVENT_ID: VM_DOWN_ERROR(119), VM HostedEngine is down with error. Exit message: resource busy: Failed to acquire lock: Lease is held by another host.
2019-03-20 09:51:12,416Z ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com', last response arrived 12659 ms ago.
2019-03-20 09:56:39,075Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [] Unable to RefreshCapabilities: NoRouteToHostException: No route to host
2019-03-20 09:56:39,077Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.NoRouteToHostException: No route to host
2019-03-20 09:56:42,086Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-57) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:42,087Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-57) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:45,096Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:45,097Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:48,115Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-45) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:48,115Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-45) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:51,126Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-49) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:51,127Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-49) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:54,135Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:54,135Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:57,145Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:57,145Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:57:00,290Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM b01-h03-r620.rhev.openstack.engineering.redhat.com command Get Host Capabilities failed: Recovering from crash or Initializing
2019-03-20 09:57:00,290Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Unable to RefreshCapabilities: VDSRecoveringException: Recovering from crash or Initializing


but then:
2019-03-20 10:27:10,106Z INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-9) [4f6604d7] EVENT_ID: HOST_REFRESHED_CAPABILITIES(606), Successfully refreshed the capabilities of host b01-h03-r620.rhev.openstack.engineering.redhat.com.
2019-03-20 10:27:20,651Z INFO  [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] Before acquiring lock in order to prevent monitoring for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' from data-center 'golden_env_mixed'
2019-03-20 10:27:20,651Z INFO  [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] Lock acquired, from now a monitoring of host will be skipped for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' from data-center 'golden_env_mixed'
2019-03-20 10:27:20,654Z INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] START, HostSetupNetworksVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, HostSetupNetworksVdsCommandParameters:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='false', bonding='false', networkName='internal_network', vdsmName='on73d84c2b703a4', nicName='em2', vlan='null', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='DHCP', ipv4Address='null', ipv4Netmask='null', ipv4Gateway='null', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='false'}), log id: 34f3981c
2019-03-20 10:27:29,704Z INFO  [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] Host setup networks finished. Lock released. Monitoring can run now for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' from data-center 'golden_env_mixed'
2019-03-20 10:27:29,974Z INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetSafeNetworkConfigVDSCommand] (default task-40) [7588b84a-cec8-48fc-ad9e-5180ca08dc4e] START, SetSafeNetworkConfigVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034'}), log id: 3fd7d378
2019-03-20 10:27:30,002Z INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-40) [7588b84a-cec8-48fc-ad9e-5180ca08dc4e] EVENT_ID: NETWORK_COMMINT_NETWORK_CHANGES(1,104), Network changes were saved on host b01-h03-r620.rhev.openstack.engineering.redhat.com



Version-Release number of selected component (if applicable):
4.3.2

How reproducible:
?

Steps to Reproduce:
1. deploy an hosted-engine cluster with two HE hosts
2. try to migrate the engine VM from one HE host to the second one from the engine web ui
3.

Actual results:
"Could not fetch data needed for VM migrate operation"

Expected results:
the engine triggers the migration

Additional info:

Comment 1 Simone Tiraboschi 2019-03-20 11:13:46 UTC
Created attachment 1546014 [details]
he_migrate.png

Comment 2 Simone Tiraboschi 2019-03-20 11:14:24 UTC
Created attachment 1546016 [details]
engine.log

Comment 3 Simone Tiraboschi 2019-03-20 11:19:11 UTC

*** This bug has been marked as a duplicate of bug 1670701 ***

Comment 4 Simone Tiraboschi 2019-03-20 11:20:58 UTC
Ok, also in this case is just a frontend issue:

DataProvider.js:35 DataProvider failed to fetch data Error: VmMigrateDataProvider: Failed to fetch target hosts
    at VmMigrateDataProvider.js:70
    at r (vendor.f6c3dc5f.js:sourcemap:46)
    at Generator._invoke (vendor.f6c3dc5f.js:sourcemap:46)
    at Generator.e.(/ovirt-engine/webadmin/anonymous function) [as next] (https://scale-hosted-engine-01.rhev.openstack.engineering.redhat.com/ovirt-engine/webadmin/plugin/ui-extensions/js/vendor.f6c3dc5f.js:46:146147)
    at r (plugin.1c5a6711.js:1886)
    at plugin.1c5a6711.js:1886


Note You need to log in before you can comment on or make changes to this bug.