Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1695567 - [downstream clone - 4.3.4] [RFE] Support VMs with VNC console on a FIPS enabled hypervisor
Summary: [downstream clone - 4.3.4] [RFE] Support VMs with VNC console on a FIPS enabl...
Keywords:
Status: POST
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 4.2.3
Hardware: All
OS: Linux
medium
medium
Target Milestone: ovirt-4.3.4
: ---
Assignee: Tomasz Barański
QA Contact: Liran Rotenberg
URL:
Whiteboard:
Depends On: 1595536
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-03 11:31 UTC by RHV Bugzilla Automation and Verification Bot
Modified: 2019-04-14 12:36 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
When a host is running in FIPS mode, VNC must use SASL authorization instead of regular passwords because of the weak algorithm inherent to the VNC protocol. In order to facilitate that process, an Ansible role 'ovirt-host-setup-vnc-sasl' is provided. It must be run manually on all FIPS hosts. The role does the following: * creates a (empty) SASL password database * prepares SASL config file for qemu * changes libvirt config file for qemu
Clone Of: 1595536
Environment:
Last Closed:
oVirt Team: Virt
Target Upstream Version:
lrotenbe: testing_plan_complete+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 95985 master ABANDONED [WIP] virt: Remove VNC password when FIPS is enabled 2019-04-03 11:32:08 UTC
oVirt gerrit 97070 master MERGED core: Don't use password auth in FIPS mode 2019-04-03 11:32:08 UTC
oVirt gerrit 97318 master MERGED ansible: Setup SASL authentication for VNC 2019-04-03 11:32:08 UTC
oVirt gerrit 97381 master MERGED virt: In FIPS mode, use SASL auth instead of qemu passwords 2019-04-03 11:32:08 UTC
oVirt gerrit 98661 ovirt-engine-4.3 MERGED ansible: Setup SASL authentication for VNC 2019-04-03 11:32:08 UTC
oVirt gerrit 99185 ovirt-engine-4.3 ABANDONED core: Don't use password auth in FIPS mode 2019-04-12 10:09:27 UTC
oVirt gerrit 99186 ovirt-4.3 POST virt: In FIPS mode, use SASL auth instead of qemu passwords 2019-04-04 14:15:49 UTC

Description RHV Bugzilla Automation and Verification Bot 2019-04-03 11:31:47 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1595536 +++
======================================================================

Description of problem:

A VM with a VNC console will fail to start on a host which is having fips enabled. It will fail with the error below.

2018-06-27 10:22:12,054+0530 ERROR (vm/e876d0c5) [virt.vm] (vmId='e876d0c5-6fa0-45e0-8a10-e44012a74f94') The vm start process failed (vm:943)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2872, in _run
    dom.createWithFlags(flags)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error: process exited while connecting to monitor: 2018-06-27T04:52:11.931930Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2018-06-27T04:52:11.973768Z qemu-kvm: -vnc 10.65.177.137:0,password: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or SASL authentication methods as an alternative

If the host is operating in "FIPS mode", the VM will be created with "-enable-fips" which will disable the VNC password authentication. So the VM will fail to start with the error above.

The VM with a spice console will work fine.


Version-Release number of selected component (if applicable):

vdsm-4.20.27.2-1.el7ev.x86_64


How reproducible:

100%

Steps to Reproduce:
1. Create a FIPS compliant host.

cat /proc/sys/crypto/fips_enabled
1

2. Start a VM with VNC graphics console on this host.
 
3. This will fail with the error as mentioned above.


Actual results:

Not possible to start a VM with VNC console on a FIPS compliant host.


Expected results:

It should be possible to start a VM with VNC console on a FIPS compliant host.

Additional info:

(Originally by Nijin Ashok)

Comment 1 RHV Bugzilla Automation and Verification Bot 2019-04-03 11:31:49 UTC
would require securing VNC first, and change authentication method form OTP to something else
Alternatively, we can ditch VNC for FIPS hosts

(Originally by michal.skrivanek)

Comment 2 RHV Bugzilla Automation and Verification Bot 2019-04-03 11:31:51 UTC
Re-targeting to 4.3.1 since it is missing a patch, an acked blocker flag, or both

(Originally by Ryan Barry)

Comment 5 RHV Bugzilla Automation and Verification Bot 2019-04-03 11:31:56 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops@redhat.comINFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops@redhat.com

(Originally by rhv-bugzilla-bot)

Comment 7 RHV Bugzilla Automation and Verification Bot 2019-04-03 11:31:59 UTC
Verification failed on:
ovirt-engine-4.3.3.1-0.1.el7.noarch
vdsm-4.30.12-1.el7ev.x86_64

Steps:
1. Enabled FIPS on the host
# yum -y install prelink dracut-fips
# prelink -u -a
# dracut -f
# df /boot
Take the Filesystem value (for example /dev/vda1 or /dev/sda1)
# blkid $filesystem
for example: # blkid /dev/sda1
Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14
Edit /etc/default/grub fips=1 and boot=$uuid:
GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....."
Regenerate grub, BIOS host:
# grub2-mkconfig -o /boot/grub2/grub.cfg
# reboot

2. Check FIPS enabled:
# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
# cat /proc/sys/crypto/fips_enabled 
1

3. Run the new ansible playbook:
Copy ssh-key:
# ssh-copy-id -i <key_path> <user>@<host>
Edit /etc/ansible/hosts
Add:
<host> ansible_ssh_private_key_file=<path>
Run:
# ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml
3. Edit a VM to VNC console.
4. Run the VM on the FIPS enabled host.

Results:
Run VM failed.
Engine log:
2019-04-02 15:30:58,045+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_DOWN_ERROR(119), VM golden_env_mixed_virtio_0 is down with error
. Exit message: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial 
NUMA mappings is obsoleted and will be removed in future
2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S
ASL authentication methods as an alternative.
2019-04-02 15:30:58,045+03 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] add VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'(golden_env_mixed_virtio_0) to rerun treatment
2019-04-02 15:30:58,051+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-11) [] Rerun VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'. Called from VDS 'host_mixed_2'
2019-04-02 15:30:58,060+03 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3309) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run 
VM golden_env_mixed_virtio_0 on Host host_mixed_2.

VDSM:
2019-04-02 15:30:57,721+0300 ERROR (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') The vm start process failed (vm:937)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 866, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2864, in _run
    dom.createWithFlags(flags)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NU
MA mappings is obsoleted and will be removed in future
2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S
ASL authentication methods as an alternative
2019-04-02 15:30:57,724+0300 INFO  (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Changed state to Down: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z q
emu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S
ASL authentication methods as an alternative (code=1) (vm:1675)
2019-04-02 15:30:57,727+0300 INFO  (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Stopping connection (guestagent:455)

Additional information:
I suspect that we miss vdsm patch on 4.3 branch: https://gerrit.ovirt.org/#/c/97381/

(Originally by Liran Rotenberg)

Comment 8 RHV Bugzilla Automation and Verification Bot 2019-04-03 11:32:01 UTC
(In reply to Liran Rotenberg from comment #7)
> I suspect that we miss vdsm patch on 4.3 branch:
> https://gerrit.ovirt.org/#/c/97381/

indeed. too late for 4.3.3 unfortunately

(Originally by michal.skrivanek)


Note You need to log in before you can comment on or make changes to this bug.