Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1348640 - HE can't get started if a new vNIC was added with an empty profile.
Summary: HE can't get started if a new vNIC was added with an empty profile.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.HostedEngine
Version: 4.0.0
Hardware: x86_64
OS: Linux
high
medium vote
Target Milestone: ovirt-4.0.2
: 4.0.2
Assignee: Jenny Tokar
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On:
Blocks: 1356483
TreeView+ depends on / blocked
 
Reported: 2016-06-21 16:00 UTC by Nikolai Sednev
Modified: 2017-05-11 09:26 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1356483 (view as bug list)
Environment:
Last Closed: 2016-08-12 14:30:18 UTC
oVirt Team: SLA
rule-engine: ovirt-4.0.z+
mgoldboi: planning_ack+
rgolan: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)
sosreport from host alma03 (deleted)
2016-06-21 16:18 UTC, Nikolai Sednev
no flags Details
sosreport from host alma04 (deleted)
2016-06-21 16:23 UTC, Nikolai Sednev
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 60258 master MERGED core: Disable adding a nic with empty profile to HE vm 2016-07-18 19:16:39 UTC
oVirt gerrit 60950 ovirt-engine-4.0 MERGED core: Disable adding a nic with empty profile to HE vm 2016-07-19 06:36:49 UTC
oVirt gerrit 60951 ovirt-engine-4.0.1 ABANDONED core: Disable adding a nic with empty profile to HE vm 2016-07-19 06:51:47 UTC

Description Nikolai Sednev 2016-06-21 16:00:15 UTC
Description of problem:
Hosted engine can't get started on host after host or HE's VM were rebooted if before their reboot a vNIC with Empty profile was added.

jsonrpc.Executor/0::INFO::2016-06-21 18:33:56,677::vm::2100::virt.vm::(hotplugNic) vmId=`24e9a52f-498f-4f76-b44c-f063a850b89f`::Hotplug NIC xml: <interface type="bridge">
        <mac address="00:1a:4a:16:01:53"/>
        <model type="virtio"/>
        <source bridge=";vdsmdummy;"/>
        <link state="up"/>
</interface>



Thread-169::DEBUG::2016-06-21 18:44:41,028::bindingxmlrpc::1242::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Down', 'exitMessage': "Cannot get interface MTU on 'None': No such device", 'statusTime': '4294965710', 'vmId': '24e9a52f-498f-4f76-b44c-f063a850b89f', 'exitReason': 1, 'exitCode': 1}]}

The regular guest VMs with the same vNIC Empty profiles receiving "vdsmdummy" instead of "none", and getting normal MTU value of 1500.

Version-Release number of selected component (if applicable):
Host:
libvirt-client-1.2.17-13.el7_2.5.x86_64
ovirt-vmconsole-1.0.3-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch
vdsm-4.18.3-0.el7ev.x86_64
ovirt-setup-lib-1.0.2-1.el7ev.noarch
ovirt-hosted-engine-ha-2.0.0-1.el7ev.noarch
mom-0.5.4-1.el7ev.noarch
sanlock-3.2.4-2.el7_2.x86_64
ovirt-host-deploy-1.5.0-1.el7ev.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.16.x86_64
ovirt-vmconsole-host-1.0.3-1.el7ev.noarch
ovirt-hosted-engine-setup-2.0.0.2-1.el7ev.noarch
Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016
Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)

Engine:
rhevm-doc-4.0.0-2.el7ev.noarch
rhev-release-4.0.0-18-001.noarch
rhevm-setup-plugins-4.0.0.1-1.el7ev.noarch
rhevm-spice-client-x64-msi-4.0-2.el7ev.noarch
rhevm-branding-rhev-4.0.0-1.el7ev.noarch
rhevm-4.0.0.6-0.1.el7ev.noarch
rhevm-guest-agent-common-1.0.12-2.el7ev.noarch
rhevm-dependencies-4.0.0-1.el7ev.noarch
rhevm-spice-client-x86-msi-4.0-2.el7ev.noarch
rhev-guest-tools-iso-4.0-2.el7ev.noarch
Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016
Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)

How reproducible:
100%

Steps to Reproduce:
1.Add new vNIC with Empty profile to HE's VM.
2.Reboot the host or the HE's VM.
3.

Actual results:
Engine could not get started any more on host.

Expected results:
Engine should get started successfully.

Additional info:
Logs from host being attached, together with screen capture of what was done.

Comment 2 Nikolai Sednev 2016-06-21 16:18:52 UTC
Created attachment 1170332 [details]
sosreport from host alma03

Comment 3 Nikolai Sednev 2016-06-21 16:23:02 UTC
Created attachment 1170333 [details]
sosreport from host alma04

Comment 4 Dan Kenigsberg 2016-06-22 07:58:42 UTC
Hosted Engine asked Vdsm to hot plug a vNIC to a non-existing network named "None". HE should send the empty string instead, if it wants to have a disconnected vNIC on the VM.

I believe that it is a clone of a bug where the empty string was serialized as "None" into the OVF, but I cannot failed to find it.

Comment 5 Nikolai Sednev 2016-06-22 08:14:26 UTC
(In reply to Dan Kenigsberg from comment #4)
> Hosted Engine asked Vdsm to hot plug a vNIC to a non-existing network named
> "None". HE should send the empty string instead, if it wants to have a
> disconnected vNIC on the VM.
> 
> I believe that it is a clone of a bug where the empty string was serialized
> as "None" into the OVF, but I cannot failed to find it.

Looks pretty familiar, isn't it this one?
https://bugzilla.redhat.com/show_bug.cgi?id=1269768

Look at the https://bugzilla.redhat.com/show_bug.cgi?id=1269768#c31

a0a7-2c0339ecd8e1`::Task._run: d1c6366c-3709-4fe9-a0a7-2c0339ecd8e1 (None, u'00000001-0001-0001-0001-000000000212', u'Default', u'97f5a165-4df5-4bce-99cc-8a634753bc54', [u'97f5a165-4df5-4bce-99cc-8a634753bc54'], 1, None, 5, 60, 10, 3) {} failed - stopping task

Comment 6 Nikolai Sednev 2016-06-22 08:18:08 UTC
I would like to add, that this is happening only on HE-VM, regular guest-VMs are not affected by this scenario, even if vNICs with an Empty profile being added to regular VMs.

Comment 7 Marina 2016-06-24 22:11:54 UTC
If a customer actually succeeds to do this (not hard, just add a nic without associating it with a profile), HE will become non-operational, until hosted-engine VM would not be started with a modified vm.conf without the problematic nic. I believe this bug needs to be set with higher priority and once we get a fix, cloned to 3.6.z as well. 

For now, I will create a kcs, but if this would happen in customer environment, it will automatically be considered sev1 issue.

Should I clone it to d/s? What would be the best next step here?

Comment 8 Roy Golan 2016-06-27 12:25:10 UTC
We should block adding a vNic with an empty profile to HE VM. 

It's worth noting that an empty profile is used in import/export of a VM where the Network of the exported nic doesn't exist in the engine setup

Comment 9 Red Hat Bugzilla Rules Engine 2016-06-27 12:25:18 UTC
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 10 Nikolai Sednev 2016-06-27 16:10:48 UTC
(In reply to Roy Golan from comment #8)
> We should block adding a vNic with an empty profile to HE VM. 
> 
> It's worth noting that an empty profile is used in import/export of a VM
> where the Network of the exported nic doesn't exist in the engine setup

IMHO you should deal with this newly added vNIC with an empty profile the same way you're dealing with it on regular guest-VMs, meaning to give it "dummyinterface" instead of giving it none/0.

Comment 17 Nikolai Sednev 2016-07-28 07:26:17 UTC
Getting an error when trying to add NIC with an empty profile:	
Operation Canceled
Error while executing action:
HostedEngine:
    Hosted Engine VM cannot have network interface with empty profile.


Works for me on these components:
Engine:
ovirt-engine-setup-plugin-ovirt-engine-common-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-websocket-proxy-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-userportal-4.0.2.1-0.1.el7ev.noarch
ovirt-iso-uploader-4.0.0-1.el7ev.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.0.2.1-0.1.el7ev.noarch
ovirt-vmconsole-proxy-1.0.4-1.el7ev.noarch
ovirt-engine-dwh-setup-4.0.2-1.el7ev.noarch
ovirt-engine-webadmin-portal-debuginfo-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-backend-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-dwh-4.0.2-1.el7ev.noarch
ovirt-engine-cli-3.6.7.0-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-setup-lib-1.0.2-1.el7ev.noarch
ovirt-engine-dashboard-1.0.1-0.el7ev.x86_64
ovirt-engine-setup-plugin-websocket-proxy-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-setup-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-extensions-api-impl-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-userportal-debuginfo-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-webadmin-portal-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-dbscripts-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-restapi-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch
ovirt-log-collector-4.0.0-1.el7ev.noarch
ovirt-host-deploy-java-1.5.1-1.el7ev.noarch
ovirt-engine-lib-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-tools-backup-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-4.0.2.1-0.1.el7ev.noarch
python-ovirt-engine-sdk4-4.0.0-0.5.a5.el7ev.x86_64
ovirt-engine-setup-base-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-vmconsole-proxy-helper-4.0.2.1-0.1.el7ev.noarch
ovirt-engine-tools-4.0.2.1-0.1.el7ev.noarch
ovirt-image-uploader-4.0.0-1.el7ev.noarch
ovirt-host-deploy-1.5.1-1.el7ev.noarch
ovirt-engine-extension-aaa-jdbc-1.1.0-1.el7ev.noarch
rhev-guest-tools-iso-4.0-5.el7ev.noarch
rhev-release-4.0.2-3-001.noarch
rhevm-4.0.2.1-0.1.el7ev.noarch
rhevm-setup-plugins-4.0.0.2-1.el7ev.noarch
rhevm-doc-4.0.0-3.el7ev.noarch
rhevm-branding-rhev-4.0.0-3.el7ev.noarch
rhevm-guest-agent-common-1.0.12-3.el7ev.noarch
rhevm-spice-client-x64-msi-4.0-3.el7ev.noarch
rhevm-spice-client-x86-msi-4.0-3.el7ev.noarch
rhevm-dependencies-4.0.0-1.el7ev.noarch
rhev-release-4.0.1-2-001.noarch
Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016
Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)


Host:
ovirt-imageio-daemon-0.3.0-0.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.5.x86_64
ovirt-vmconsole-1.0.4-1.el7ev.noarch
vdsm-4.18.9-1.el7ev.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.19.x86_64
ovirt-host-deploy-1.5.1-1.el7ev.noarch
ovirt-hosted-engine-ha-2.0.1-1.el7ev.noarch
ovirt-hosted-engine-setup-2.0.1.3-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch
mom-0.5.5-1.el7ev.noarch
ovirt-setup-lib-1.0.2-1.el7ev.noarch
ovirt-imageio-common-0.3.0-0.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
rhev-release-4.0.2-3-001.noarch
sanlock-3.2.4-3.el7_2.x86_64
Linux version 3.10.0-327.30.1.el7.x86_64 (mockbuild@x86-039.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Wed Jul 13 22:09:46 EDT 2016
Linux 3.10.0-327.30.1.el7.x86_64 #1 SMP Wed Jul 13 22:09:46 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)


Note You need to log in before you can comment on or make changes to this bug.