Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1357246 - Hosted Engine - Not enough entropy
Summary: Hosted Engine - Not enough entropy
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Plugins.General
Version: ---
Hardware: Unspecified
OS: Unspecified
high
high vote
Target Milestone: ovirt-4.0.5
: ---
Assignee: Roy Golan
QA Contact: Artyom
URL:
Whiteboard:
Depends On:
Blocks: 1352384
TreeView+ depends on / blocked
 
Reported: 2016-07-17 08:00 UTC by Yedidyah Bar David
Modified: 2017-05-11 09:27 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-18 07:35:57 UTC
oVirt Team: Integration
rule-engine: ovirt-4.0.z+
rule-engine: planning_ack+
sbonazzo: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)
host and vm entroy (deleted)
2016-09-21 06:16 UTC, Artyom
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 62333 master MERGED Add rng device to bootsrap vm.conf 2016-08-19 15:09:39 UTC
oVirt gerrit 63467 ovirt-hosted-engine-setup-2.0 MERGED Add rng device to bootsrap vm.conf 2016-09-07 07:44:54 UTC
oVirt gerrit 63703 master MERGED random: Add rng device to initial VM creation 2016-09-14 07:48:57 UTC
oVirt gerrit 63763 ovirt-hosted-engine-setup-2.0 MERGED random: Add rng device to initial VM creation 2016-09-14 11:24:59 UTC
Red Hat Bugzilla 1368027 None CLOSED Hosted Engine VM needs RNG for entropy 2019-04-05 00:02:31 UTC

Internal Links: 1368027

Description Yedidyah Bar David 2016-07-17 08:00:05 UTC
Description of problem:

See e.g. bug 1352384.

We should create the engine vm with better entropy sources, e.g. virtio-rng or haveged.

If possible, it's probably better to do that in the appliance, so that it's applicable in every installation based on it.

Please clone or move to hosted-engine-setup if needed. Not sure.

Version-Release number of selected component (if applicable):

Current

How reproducible:

Always

Steps to Reproduce:
1. Deploy a machine based on the engine appliance
2. cat /proc/sys/kernel/random/entropy_avail
3. Optionally "use" some random data, e.g.:
sh -xc 'while true; do dd if=/dev/random of=/dev/null bs=1 count=5; cat /proc/sys/kernel/random/entropy_avail; sleep 1; done'

Actual results:

Entropy is gathered slowly

Expected results:

Entropy is gathered faster

Additional info:

As per linked bug, a useful actual metric for success is time for engine-setup completion. I'd say it should be no more than a minute or so more that the time it takes on the physical host.

Comment 1 Fabian Deutsch 2016-07-17 15:21:12 UTC
Didi, adding virtio-rng sounds reasonable. But this would be something which should be done by he-setup.

I do not think that we can get better entropy by changing the appliance.

Comment 2 Doron Fediuck 2016-08-21 07:42:14 UTC
*** Bug 1368027 has been marked as a duplicate of this bug. ***

Comment 3 Artyom 2016-09-20 14:30:51 UTC
Checked on ovirt-hosted-engine-setup-2.0.2.2-2.el7ev.noarch

I can see that HE VM has rng device:
<rng model='virtio'>
      <backend model='random'>/dev/random</backend>
      <alias name='rng0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </rng>

but it does not raise entropy level
# time sh -xc 'while true; do dd if=/dev/random of=/dev/null bs=1 count=5; cat /proc/sys/kernel/random/entropy_avail; sleep 1; done' 
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 20.9678 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
153
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000109729 s, 45.6 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
90
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000126877 s, 39.4 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
27
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 37.5261 s, 0.0 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
153
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000120944 s, 41.3 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
89
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000107812 s, 46.4 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
91
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000108143 s, 46.2 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
28
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
^C1+0 records in
1+0 records out
1 byte (1 B) copied, 16.0376 s, 0.0 kB/s


real    1m21.553s
user    0m0.007s
sys     0m0.015s

I believe HE VM just does not run enough time with enough devices, so maybe we can use something like rng-tools to generate the desired level of the entropy

Comment 4 Simone Tiraboschi 2016-09-20 14:51:54 UTC
Can you please run
 cat /sys/devices/virtual/misc/hw_random/rng_current
on your engine vm?

Comment 5 Artyom 2016-09-20 14:56:28 UTC
Sure
# cat /sys/devices/virtual/misc/hw_random/rng_current
virtio_rng.0

I checked entropy level after rng-tools install:
# yum install -y rng-tools
# systemctl start rngd
# time sh -xc 'while true; do dd if=/dev/random of=/dev/null bs=1 count=5; cat /proc/sys/kernel/random/entropy_avail; sleep 1; done'
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000153841 s, 32.5 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
3167
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000143667 s, 34.8 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
3119
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000133049 s, 37.6 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
2982
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000144017 s, 34.7 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
3038
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000114122 s, 43.8 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
2995
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000131049 s, 38.2 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
3025
+ sleep 1
+ true
+ dd if=/dev/random of=/dev/null bs=1 count=5
5+0 records in
5+0 records out
5 bytes (5 B) copied, 0.000130214 s, 38.4 kB/s
+ cat /proc/sys/kernel/random/entropy_avail
2577
+ sleep 1
^C

real    0m6.398s
user    0m0.006s
sys     0m0.017s

Comment 6 Yedidyah Bar David 2016-09-20 15:08:23 UTC
(In reply to Artyom from comment #3)
> Checked on ovirt-hosted-engine-setup-2.0.2.2-2.el7ev.noarch
> 
> I can see that HE VM has rng device:
> <rng model='virtio'>
>       <backend model='random'>/dev/random</backend>
>       <alias name='rng0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </rng>
> 
> but it does not raise entropy level

Why do you think so?

> # time sh -xc 'while true; do dd if=/dev/random of=/dev/null bs=1 count=5;
> cat /proc/sys/kernel/random/entropy_avail; sleep 1; done' 
> + true
> + dd if=/dev/random of=/dev/null bs=1 count=5
> 5+0 records in
> 5+0 records out
> 5 bytes (5 B) copied, 20.9678 s, 0.0 kB/s
> + cat /proc/sys/kernel/random/entropy_avail
> 153
> + sleep 1
> + true
> + dd if=/dev/random of=/dev/null bs=1 count=5
> 5+0 records in
> 5+0 records out
> 5 bytes (5 B) copied, 0.000109729 s, 45.6 kB/s
> + cat /proc/sys/kernel/random/entropy_avail
> 90
> + sleep 1
> + true
> + dd if=/dev/random of=/dev/null bs=1 count=5
> 5+0 records in
> 5+0 records out
> 5 bytes (5 B) copied, 0.000126877 s, 39.4 kB/s
> + cat /proc/sys/kernel/random/entropy_avail
> 27
> + sleep 1
> + true
> + dd if=/dev/random of=/dev/null bs=1 count=5
> 5+0 records in
> 5+0 records out
> 5 bytes (5 B) copied, 37.5261 s, 0.0 kB/s
> + cat /proc/sys/kernel/random/entropy_avail
> 153

Seems it raised it

> + sleep 1
> + true
> + dd if=/dev/random of=/dev/null bs=1 count=5
> 5+0 records in
> 5+0 records out
> 5 bytes (5 B) copied, 0.000120944 s, 41.3 kB/s
> + cat /proc/sys/kernel/random/entropy_avail
> 89
> + sleep 1
> + true
> + dd if=/dev/random of=/dev/null bs=1 count=5
> 5+0 records in
> 5+0 records out
> 5 bytes (5 B) copied, 0.000107812 s, 46.4 kB/s
> + cat /proc/sys/kernel/random/entropy_avail
> 91
> + sleep 1
> + true
> + dd if=/dev/random of=/dev/null bs=1 count=5
> 5+0 records in
> 5+0 records out
> 5 bytes (5 B) copied, 0.000108143 s, 46.2 kB/s
> + cat /proc/sys/kernel/random/entropy_avail
> 28
> + sleep 1
> + true
> + dd if=/dev/random of=/dev/null bs=1 count=5
> ^C1+0 records in
> 1+0 records out
> 1 byte (1 B) copied, 16.0376 s, 0.0 kB/s

Perhaps let it run for some more time, and also:

1. Compare with unpatched version
2. Compare with the host it's running on

> 
> 
> real    1m21.553s
> user    0m0.007s
> sys     0m0.015s
> 
> I believe HE VM just does not run enough time with enough devices, so maybe
> we can use something like rng-tools to generate the desired level of the
> entropy

I'd not do that without consulting a security expert, and/or making it optional (so that users that care can override).

Comment 7 Simone Tiraboschi 2016-09-20 15:09:41 UTC
(In reply to Artyom from comment #5)
> Sure
> # cat /sys/devices/virtual/misc/hw_random/rng_current
> virtio_rng.0
> 
> I checked entropy level after rng-tools install:
> # yum install -y rng-tools
> # systemctl start rngd

According to http://rhelblog.redhat.com/2015/03/09/red-hat-enterprise-linux-virtual-machines-access-to-random-numbers-made-easy/ there is no need to run rng-tools since 7.1 and it should work transparently.

Comment 8 Simone Tiraboschi 2016-09-20 15:51:23 UTC
It seams that now the same VM is fine on the same test also removing the  rng-tools package.
I suspect that the host entropy was low during the test at https://bugzilla.redhat.com/show_bug.cgi?id=1357246#c3

Comment 9 Red Hat Bugzilla Rules Engine 2016-09-20 17:55:11 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 10 Artyom 2016-09-21 06:16:53 UTC
Created attachment 1203111 [details]
host and vm entroy

I restarted HE VM(on deploy time, so VM still has virtio_rng.0 device), and checked entropy level for the VM and for the host where VM is running for the long period of time and for me it looks like passthrough of random dev to the VM does not work at all

Comment 11 Simone Tiraboschi 2016-09-21 11:39:53 UTC
(In reply to Simone Tiraboschi from comment #7)
> According to
> http://rhelblog.redhat.com/2015/03/09/red-hat-enterprise-linux-virtual-
> machines-access-to-random-numbers-made-easy/ there is no need to run
> rng-tools since 7.1 and it should work transparently.

Amit, do we still have to start the rngd daemon on a RHEL 7.2 guest to take advantage of virtio-rng?

Comment 12 Amit Shah 2016-09-21 11:58:34 UTC
(In reply to Simone Tiraboschi from comment #11)
> (In reply to Simone Tiraboschi from comment #7)
> > According to
> > http://rhelblog.redhat.com/2015/03/09/red-hat-enterprise-linux-virtual-
> > machines-access-to-random-numbers-made-easy/ there is no need to run
> > rng-tools since 7.1 and it should work transparently.
> 
> Amit, do we still have to start the rngd daemon on a RHEL 7.2 guest to take
> advantage of virtio-rng?

For just virtio-rng, rngd is not needed.  All logic is handled in the kernel.

Comment 13 Simone Tiraboschi 2016-09-21 12:47:07 UTC
(In reply to Amit Shah from comment #12)
> For just virtio-rng, rngd is not needed.  All logic is handled in the kernel.

Thanks Amir,
According to https://bugzilla.redhat.com/show_bug.cgi?id=1357246#c10 it seams that we are not fully satisfied of the results on the guest while the host deams fine.

Could it depends from virtio-rng configuration?
We are just relying on the default values without specifying a different bytes and period rate.

The device looks like:
<rng model='virtio'>
   <backend model='random'>/dev/random</backend>
   <alias name='rng0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</rng>

Comment 14 Amit Shah 2016-09-22 08:16:43 UTC
I guess the test in question just reads from /dev/random in the guest?  That's triggering the worst-case behaviour: continuously requesting entropy from the host.  Real workloads are not going to put such pressure.

I suggest you try a more real-world test, like bug 1352384 mentioned in comment 0.

It's also OK to use /dev/urandom as source (instead of /dev/random) in both the guest and the host.  Reading from /dev/urandom doesn't affect the quality of randomness you get.

Comment 15 Simone Tiraboschi 2016-09-22 09:06:46 UTC
According to comment https://bugzilla.redhat.com/show_bug.cgi?id=1357246#c14
I'd suggest to verify this bug simply measuring how long does it take to setup hosted-engine with and without these patches on the same HW and storage.
If we got a significative improvement we can set it as verified.

Comment 16 Simone Tiraboschi 2016-09-22 09:12:56 UTC
So ovirt-hosted-engine-setup-2.0.2 vs ovirt-hosted-engine-setup-2.0.2.2 without changing any other package.

Comment 17 Artyom 2016-09-22 13:40:54 UTC
From the execution of the HE with the different packages, I can see improvement:
ovirt-hosted-engine-setup-2.0.2.2-2.el7ev.noarch
===============================================
[ INFO  ] Hosted Engine successfully deployed

real    12m4.439s
user    2m40.931s
sys     0m11.821s

14:13:16 - Running engine-setup on the appliance
14:15:48 - Engine-setup successfully completed


ovirt-hosted-engine-setup-2.0.2.1-1.el7ev.noarch
===============================================
[ INFO  ] Hosted Engine successfully deployed

real    18m7.475s
user    2m41.469s
sys     0m12.059s

16:14:20 - Running engine-setup on the appliance
16:21:02 - Engine-setup successfully completed

Verified on ovirt-hosted-engine-setup-2.0.2.1-1.el7ev.noarch


Note You need to log in before you can comment on or make changes to this bug.