Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1368027 - Hosted Engine VM needs RNG for entropy
Summary: Hosted Engine VM needs RNG for entropy
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-hosted-engine-ha
Classification: oVirt
Component: Agent
Version: 2.0.0
Hardware: Unspecified
OS: Unspecified
high
high vote
Target Milestone: ovirt-4.1.0-beta
: 2.1.0
Assignee: Jenny Tokar
QA Contact: Nikolai Sednev
URL:
Whiteboard: PM-08
: 1417176 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-18 07:25 UTC by Roy Golan
Modified: 2017-02-15 15:08 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The vm configuration was created without rng device. Consequence: The vm had little entropy which caused long wait times. Fix: Use the rng device that was provided in the ovf, if no device was provided create a stubbed one. Result: The vm has enough entropy and the used configuration is correct.
Clone Of:
Environment:
Last Closed: 2017-02-15 15:08:09 UTC
oVirt Team: SLA
rule-engine: ovirt-4.1+
mgoldboi: exception+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 62334 master ABANDONED Stub the rng device on the ovf convert 2017-02-12 17:59:30 UTC
oVirt gerrit 69234 master MERGED Use RNG device from OVF if provided 2017-01-05 09:47:53 UTC
oVirt gerrit 69694 v2.1.z MERGED Use RNG device from OVF if provided 2017-01-09 16:05:35 UTC
oVirt gerrit 69923 v2.1.z MERGED Fix specParams parsing in OVF converter and update the RNG code (this fixed the latest build failure) 2017-01-10 16:42:01 UTC
Red Hat Bugzilla 1357246 None CLOSED Hosted Engine - Not enough entropy 2019-04-05 00:05:33 UTC
Red Hat Bugzilla 1364132 None CLOSED Once the engine imports the hosted-engine VM we lose the console device 2019-04-05 00:05:33 UTC

Internal Links: 1357246 1364132

Description Roy Golan 2016-08-18 07:25:50 UTC
Description of problem:

Hosted Engine VM have very little entropy available which makes the setup and SSL logins waiting for long time.

Version-Release number of selected component (if applicable):
all

How reproducible:
100%

Steps to Reproduce:
1. Deploy a machine based on the engine appliance
2. cat /proc/sys/kernel/random/entropy_avail
3. Optionally "use" some random data, e.g.:
sh -xc 'while true; do dd if=/dev/random of=/dev/null bs=1 count=5; cat /proc/sys/kernel/random/entropy_avail; sleep 1; done'
Steps to Reproduce:

Actual results:

Very little entropy

Expected results:

Enough entropy

Additional info:

Comment 1 Yedidyah Bar David 2016-08-18 11:24:35 UTC
Clone of bug 1357246 ?

Comment 3 Roy Golan 2016-09-07 06:59:06 UTC
this is not a duplicate because one is ha agent and Bug 1357246 is for the setup

Comment 4 Yaniv Kaul 2016-10-25 08:24:45 UTC
Is it going to 4.0.5 or not?

Comment 5 Yaniv Kaul 2016-11-29 08:28:47 UTC
(In reply to Yaniv Kaul from comment #4)
> Is it going to 4.0.5 or not?

4.0.6?

Comment 6 Roy Golan 2016-11-29 12:36:37 UTC
(In reply to Yaniv Kaul from comment #5)
> (In reply to Yaniv Kaul from comment #4)
> > Is it going to 4.0.5 or not?
> 
> 4.0.6?

yes. Engine is going to add the device to the VM always in 4.0. an import of the VM in 4.1 will have the device by default because of blank template.
Another patch will be sent to ovirt-ha-agent to handle rng device conversion from OVF to vm.conf

Comment 7 Michal Skrivanek 2016-11-29 13:31:58 UTC
I do NOT recommend 4.0.z due to the introduction of virtio-rng by default and a change to urandom source. You would save yourself a hassle of testing and potential problems during 4.1 HE upgrade. 
Unless you want to verify the flows and check the behavior, there are slightly different code paths taken for regulat VMs and HE and it may break.

Comment 9 Doron Fediuck 2016-12-05 13:30:25 UTC
This will be resolved in 4.1 (already merged in master).
For 4.0.z we have no capacity to handle it.

Comment 10 Nirav Dave 2017-02-03 16:45:56 UTC
*** Bug 1417176 has been marked as a duplicate of this bug. ***

Comment 11 Nikolai Sednev 2017-02-08 22:30:33 UTC
It doesn't looks like there is enough entropy on HE-VM: 

cat /run/ovirt-hosted-engine-ha/vm.conf | grep random
devices={device:virtio,specParams:{source:random},model:virtio,type:rng}

cat /proc/sys/kernel/random/entropy_avail
151

# time sh -xc 'while true; do dd if=/dev/random of=/dev/null bs=1 count=5; cat /proc/sys/kernel/random/entropy_avail; sleep 1; done' 
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000182089 s, 27.5 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
61
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 2.41893 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
94
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.00022185 s, 22.5 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
31
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 33.5312 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
99
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000195219 s, 25.6 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
37
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000217526 s, 23.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
38
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
^C0+0 records in
0+0 records out
0 bytes (0 B) copied, 2.54474 s, 0.0 kB/s


real    0m44.523s
user    0m0.008s
sys     0m0.021s


# time sh -xc 'while true; do dd if=/dev/random of=/dev/null bs=1 count=5; cat /proc/sys/kernel/random/entropy_avail; sleep 1; done' 
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000182068 s, 27.5 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
23
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 39.3871 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
94
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000198586 s, 25.2 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
30
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 33.2879 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
76
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000182515 s, 27.4 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
105
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000177201 s, 28.2 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
106
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000182921 s, 27.3 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
44
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 20.4807 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
34
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 25.0991 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
93
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000193267 s, 25.9 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
29
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 32.9122 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
69
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000221004 s, 22.6 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
70
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000227211 s, 22.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
106
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000184224 s, 27.1 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
44
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 17.5625 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
35
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
^C3+0 records in
3+0 records out
3 bytes (3 B) copied, 1.67787 s, 0.0 kB/s


real    3m5.470s
user    0m0.016s
sys     0m0.050s

Comment 12 Nikolai Sednev 2017-02-08 22:35:27 UTC
Here https://bugzilla.redhat.com/show_bug.cgi?id=1357246#c5 entropy looks much better.

Tested on 4.1RHEVH7.3:
rhvm-appliance-4.1.20170126.0-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-hosted-engine-ha-2.1.0.1-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.0.1-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-node-ng-nodectl-4.1.0-0.20170104.1.el7.noarch
libvirt-client-2.0.0-10.el7_3.4.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64
vdsm-4.19.4-1.el7ev.x86_64
sanlock-3.4.0-1.el7.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
mom-0.5.8-1.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 7.3

Engine was also updated to latest available components:
rhevm-setup-plugins-4.1.0-1.el7ev.noarch
rhevm-dependencies-4.1.0-1.el7ev.noarch
rhevm-doc-4.1.0-2.el7ev.noarch
rhev-guest-tools-iso-4.1-3.el7ev.noarch
rhevm-4.1.0.4-0.1.el7.noarch
rhevm-branding-rhev-4.1.0-0.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

# cat /run/ovirt-hosted-engine-ha/vm.conf 
vmId=f8f9a460-ae38-49df-b14e-0177fb1ce928
memSize=16384
display=vnc
devices={index:2,iface:ide,address:{ controller:0, target:0,unit:0, bus:1, type:drive},specParams:{},readonly:true,deviceId:17f40ffd-ead3-4874-a49a-e610680772e6,path:,device:cdrom,shared:false,type:disk}
devices={index:0,iface:virtio,format:raw,poolID:00000000-0000-0000-0000-000000000000,volumeID:4aa67d26-ad22-41ec-8a67-1dfe04908c2d,imageID:e496ec13-6f14-44f6-9a5a-b0896035602a,specParams:{},readonly:false,domainID:e2f5c816-4b39-4819-9bbe-c02298841c02,optional:false,deviceId:e496ec13-6f14-44f6-9a5a-b0896035602a,address:{bus:0x00, slot:0x06, domain:0x0000, type:pci, function:0x0},device:disk,shared:exclusive,propagateErrors:off,type:disk,bootOrder:1}
devices={device:scsi,model:virtio-scsi,type:controller}
devices={nicModel:pv,macAddr:00:16:3E:7B:B8:53,linkActive:true,network:ovirtmgmt,specParams:{},deviceId:bc8e2aae-6676-4f34-9494-a218999e83b1,address:{bus:0x00, slot:0x03, domain:0x0000, type:pci, function:0x0},device:bridge,type:interface}
devices={device:console,specParams:{},type:console,deviceId:3671bd23-f029-4ac6-bf7d-8f49a48be23b,alias:console0}
devices={device:vga,alias:video0,type:video}
devices={device:vnc,type:graphics}
vmName=HostedEngine
spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir
smp=4
maxVCpus=4
cpuType=SandyBridge
emulatedMachine=pc-i440fx-rhel7.3.0
devices={device:virtio,specParams:{source:random},model:virtio,type:rng}

Comment 13 Nikolai Sednev 2017-02-08 22:36:19 UTC
Roy, may you please provide your input on this?

Comment 14 Roy Golan 2017-02-12 14:18:45 UTC
The scope of this change was to charge the hosted engine vm with entropy from the host pool. It's good that the hevm is started with the rng device (by initial setup and by consecutive boot by the ha agent and with the vm.conf from shared storage)

If the host if falling short with entropy that might be because of a small scale setup with drivers not working hard enough. There are ways to enhance that and I think you suggested something on IRC, please share it here if relevant.

Comment 15 Nikolai Sednev 2017-02-12 14:59:22 UTC
Moving to verified forth to bug's scope and these details:
alma04 ~]# virsh -r dumpxml HostedEngine | grep rng
    <rng model='virtio'>
      <alias name='rng0'/>
    </rng>
[root@alma04 ~]#  cat /proc/sys/kernel/random/entropy_avail
153
[root@alma04 ~]#  virsh -r list
 Id    Name                           State
----------------------------------------------------
 1     HostedEngine                   running
cat /run/ovirt-hosted-engine-ha/vm.conf | grep random
devices={device:virtio,specParams:{source:random},model:virtio,type:rng}

IMHO we should enhance the entropy as its too small for VMs. I propose using rng-tools for getting better results as was proposed in https://bugzilla.redhat.com/show_bug.cgi?id=1357246#c3, as I do not see any of those results on my hosts running these components:
rhvm-appliance-4.1.20170126.0-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-hosted-engine-ha-2.1.0.1-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.0.1-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-node-ng-nodectl-4.1.0-0.20170104.1.el7.noarch
libvirt-client-2.0.0-10.el7_3.4.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64
vdsm-4.19.4-1.el7ev.x86_64
sanlock-3.4.0-1.el7.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
mom-0.5.8-1.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 7.3

Engine:
rhevm-setup-plugins-4.1.0-1.el7ev.noarch
rhevm-dependencies-4.1.0-1.el7ev.noarch
rhevm-doc-4.1.0-2.el7ev.noarch
rhev-guest-tools-iso-4.1-3.el7ev.noarch
rhevm-4.1.0.4-0.1.el7.noarch
rhevm-branding-rhev-4.1.0-0.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Looks like we're not working transparently forth to https://bugzilla.redhat.com/show_bug.cgi?id=1357246#c7.

One of the ways to check for good entropy would be to use dieharder and verify the quality of entropy in VM's pool.

I've opened an RFE to enhance our current entropy for VMs here https://bugzilla.redhat.com/show_bug.cgi?id=1421472.


Note You need to log in before you can comment on or make changes to this bug.