Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1411491 - Disk image upload fails on self-hosted engine, requires imageio-daemon restart
Summary: Disk image upload fails on self-hosted engine, requires imageio-daemon restart
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-host-deploy
Classification: oVirt
Component: Core
Version: ---
Hardware: x86_64
OS: Unspecified
unspecified
high vote
Target Milestone: ovirt-4.1.3
: 1.6.6
Assignee: Daniel Erez
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On:
Blocks: RHEV-RHCI 1418343
TreeView+ depends on / blocked
 
Reported: 2017-01-09 20:28 UTC by Derek Whatley
Modified: 2017-07-06 14:02 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-06 14:02:31 UTC
oVirt Team: Storage
rule-engine: ovirt-4.1+


Attachments (Terms of Use)
/var/log/ovirt-imageio-proxy/image-proxy.log (deleted)
2017-01-09 20:28 UTC, Derek Whatley
no flags Details
Collection of logs from hypervisor and engine. (deleted)
2017-01-11 18:35 UTC, Derek Whatley
no flags Details
logs: setup, agent, broker, daemon, vdsm and more... (deleted)
2017-03-22 17:13 UTC, Natalie Gavrielov
no flags Details
image proxy, engine, server (deleted)
2017-03-23 10:25 UTC, Natalie Gavrielov
no flags Details
host-deploy logs (deleted)
2017-03-23 11:15 UTC, Natalie Gavrielov
no flags Details
Screenshot from 2017-04-04 17-38-20.png (deleted)
2017-04-04 14:38 UTC, Nikolai Sednev
no flags Details
sosreport-nsednev-he-4.scl.lab.tlv.redhat.com-20170404173939.tar.xz (deleted)
2017-04-04 14:44 UTC, Nikolai Sednev
no flags Details
sosreport-puma18.scl.lab.tlv.redhat.com-20170404173912.tar.xz (deleted)
2017-04-04 14:45 UTC, Nikolai Sednev
no flags Details
sosreport-puma18.scl.lab.tlv.redhat.com-20170404173912.tar.xz (deleted)
2017-04-04 14:46 UTC, Nikolai Sednev
no flags Details
sosreport-puma19.scl.lab.tlv.redhat.com-20170404173929.tar.xz (deleted)
2017-04-04 14:49 UTC, Nikolai Sednev
no flags Details
screencast (deleted)
2017-06-19 12:25 UTC, Nikolai Sednev
no flags Details
sosreport from the engine (deleted)
2017-06-19 12:28 UTC, Nikolai Sednev
no flags Details
sosreport from puma18 the host (deleted)
2017-06-19 12:29 UTC, Nikolai Sednev
no flags Details
ovirt-imageio-daemon daemon.log (deleted)
2017-06-19 12:46 UTC, Nikolai Sednev
no flags Details
imageproxy log from an engine (deleted)
2017-06-19 14:52 UTC, Nikolai Sednev
no flags Details
image-proxy.log.2017-06-18 from an engine (deleted)
2017-06-19 14:53 UTC, Nikolai Sednev
no flags Details
screencast (deleted)
2017-06-19 15:19 UTC, Nikolai Sednev
no flags Details
Screenshot from 2017-06-21 14-24-42.png (deleted)
2017-06-21 11:25 UTC, Nikolai Sednev
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 71378 master MERGED vdsm: restart imageio daemon on vdsm start 2017-01-31 11:53:08 UTC
oVirt gerrit 71475 ovirt-host-deploy-1.6 MERGED vdsm: restart imageio daemon on vdsm start 2017-02-01 15:10:48 UTC

Description Derek Whatley 2017-01-09 20:28:49 UTC
Created attachment 1238880 [details]
/var/log/ovirt-imageio-proxy/image-proxy.log

=======================
Description of problem
=======================
After completing self-hosted engine setup, we attempt to upload a ~600MB qcow2 disk image via API. This image upload attempt will fail and cause the RHV Engine WebUI to enter a broken state _unless_ we first restart ovirt-imageio-daemon.service on the hypervisor hosting the engine.

When the WebUI is broken as described above, attempts at page navigation (specifically navigation to certain tabs, such as "Virtual Machines" at https://kbtest-engine.example.com/ovirt-engine/webadmin/?locale=en_US#vms) result in many "Operation Canceled" dialog boxes stacking on top of eachother with messages:
 
>>> "Error while executing action: A Request to the Server failed: java.lang.reflect.InvocationTargetException"

and 
 
>>> "Error while executing query: null"

Our image upload routine is based on RHV documentation.
https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/single/rest-api-guide/#services-image_transfer

We created an image upload script by referencing the documentation above. https://github.com/fabianvf/ansible-ovirt/blob/vm-launch/roles/launch_vms/files/upload_image#L72 

See image_proxy.log (attached, sourced from /var/log/ovirt-imageio-proxy/image-proxy.log) for the error messages we saw on the self-hosted engine while trying to upload images via API.

After restarting ovirt-imageio-daemon.service on the hypervisor, the image uploads could be completed successfully, but WebUI error messages mentioned above continued to manifest.

=============================================================
Version-Release number of selected component (if applicable)
=============================================================
Using the 4.0.6 build at http://bob.eng.lab.tlv.redhat.com/builds/latest_4.0/el7Server/

The RPM version installed is 4.0.6.2


=================
How reproducible
=================
Reproduced several times on our bare-metal and nested-virt setups within the deployment flow of the Quickstart Cloud Installer.

===================
Steps to Reproduce
===================
1. Create self-hosted RHV engine on RHV hypervisor
2. Attempt to upload disk image via API linked above
3. Monitor from RHV WebUI. Image upload should stall/fail
4. Navigate to "Virtual Machines" tab in RHV WebUI
5. You should see many error dialog boxes pop-up
--- 
6. Restart "ovirt-imageio-daemon.service" on RHV hypervisor
7. Attempt image upload again, should succeed
8. WebUI errors may still be triggered by loading certain pages
9. WebUI errors should go away if you restart RHV engine VM

=================
Expected results
=================
1. Create self-hosted RHV engine on RHV hypervisor
2. Attempt to upload disk image via API linked above
3. Image upload successful, no WebUI errors present

Comment 1 Daniel Erez 2017-01-10 08:35:10 UTC
According to the logs [1], it's an SSL issue in the daemon. When restarting the daemon the certificate reloads, hence works as expected.

@Derek - is it a completely clean installation? fresh engine and vdsm?

@Nir - can we automate the reload of the daemon's certificate? or it requires a daemon restart?

[1]
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
(Thread-24 ) ERROR 2017-01-04 14:31:04,710 web:84:web:(log_response) 192.168.200.213 - PUT  503 209 (0.01s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 43, in __call__
    resp = self.dispatch(request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 68, in dispatch
    return method(*match.groups())
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 88, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 59, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 75, in put
    return self.send_data(self.request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 116, in send_data
    request.method, imaged_url, headers, body, stream)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_handler.py", line 187, in make_imaged_request
    raise exc.HTTPServiceUnavailable(s)
HTTPServiceUnavailable: Failed communicating with vdsm-imaged: An SSL error occurred.

Comment 2 Derek Whatley 2017-01-10 17:18:23 UTC
(In reply to Daniel Erez from comment #1)
> According to the logs [1], it's an SSL issue in the daemon. When restarting
> the daemon the certificate reloads, hence works as expected.
> 
> @Derek - is it a completely clean installation? fresh engine and vdsm?

Yes, completely clean installation. The sequence is as follows:

1) Run "hosted-engine --deploy" to deploy engine and hypervisor (docs reference linked below)
2) Create disks
3) Attempt disk image upload, fails


https://access.redhat.com/documentation/en/red-hat-virtualization/4.0/paged/self-hosted-engine-guide/chapter-2-deploying-self-hosted-engine

Comment 3 Daniel Erez 2017-01-11 12:16:40 UTC
@Nir - another duplicate of bug 1401901?

Comment 4 Nir Soffer 2017-01-11 12:22:16 UTC
(In reply to Daniel Erez from comment #3)
> @Nir - another duplicate of bug 1401901?

Do you see the same error in the log? ovirt-imageio-daemon fail to start with 
permission error?

Comment 5 Daniel Erez 2017-01-11 13:48:18 UTC
@Derek - can you please also attach ovirt-imageio-daemon / vdsm / engine logs

Comment 6 Derek Whatley 2017-01-11 18:35:37 UTC
Created attachment 1239571 [details]
Collection of logs from hypervisor and engine.

If you have any specific additional requests for log files, please let me know.

Comment 7 Daniel Erez 2017-01-16 10:33:23 UTC
@Derek - seems that you using an old version without logs for imageio. Which version of imageio daemon are you using? Can you try the described scenario with the latest one?

Comment 8 Derek Whatley 2017-01-18 18:59:47 UTC
I'm running ovirt-imageio-daemon-0.4.0-0.el7ev.noarch on the hypervisor. This seems to be the most up-to-date version according to the access.redhat.com package search.

What is the name/path of the log file you were expecting to be included?

Comment 9 Fabian von Feilitzsch 2017-02-02 14:48:29 UTC
We still seem to be seeing this when deploying multiple hypervisors. When we have a self-hosted engine deployment with multiple hypervisors, we have to restart the imageio-daemon on all of them before image uploads work, not just the one hosting the engine.

Comment 10 Nir Soffer 2017-02-02 14:55:20 UTC
(In reply to Fabian von Feilitzsch from comment #9)
> We still seem to be seeing this when deploying multiple hypervisors. When we
> have a self-hosted engine deployment with multiple hypervisors, we have to
> restart the imageio-daemon on all of them before image uploads work, not
> just the one hosting the engine.

This is expected, you should restart ovirt-imageio-daemon after enrolling
certificates on a host. The patch Daniel submitted should fix this issue during
host-deploy.

Comment 11 Natalie Gavrielov 2017-03-20 17:52:13 UTC
Does it matter much if verification process will use hosted-engine on RHEL7.3? (instead of using RHV hypervisior)

Comment 12 Daniel Erez 2017-03-22 10:30:42 UTC
(In reply to Natalie Gavrielov from comment #11)
> Does it matter much if verification process will use hosted-engine on
> RHEL7.3? (instead of using RHV hypervisior)

Shouldn't matter, the flow is similar.

Comment 13 Natalie Gavrielov 2017-03-22 17:10:20 UTC
Scenario performed:
-------------------

1. On a clean RHV hypervisor, performed hosted engine installation.
2. Configured certificates for both Chrome and Firefox, with all required (and 
   possible) permissions. 
3. Tried to upload an image via GUI (disks tab -> upload)
   * Tried both RAW and Qcow2 v3 formats.

Result:
-------

All attempts to upload failed (switching status to "Paused by the system").


Important notes:
----------------
 
1. After manually restarting the ovirt-imageio-daemon upload image works!
2. I looked for the restart of the ovirt-imageio-daemon, that host deploy should do in ovirt-imageio-daemon.log and couldn't find any evidence for this restart (only the one I performed manually).
3. ovirt-imageio-proxy was not installed (never worked with hosted engine before but I would guess it's not supposed to be installed..)

Builds used:
RHVM 4.1.1.2-0.1.el7
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch

Will attach logs in no time.

Comment 14 Red Hat Bugzilla Rules Engine 2017-03-22 17:10:28 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 15 Natalie Gavrielov 2017-03-22 17:13:40 UTC
Created attachment 1265457 [details]
logs: setup, agent, broker, daemon, vdsm and more...

Comment 16 Daniel Erez 2017-03-22 20:09:34 UTC
@Natalie - which version of ovirt-host-deploy is installed?

Comment 17 Natalie Gavrielov 2017-03-23 06:55:55 UTC
(In reply to Daniel Erez from comment #16)
> @Natalie - which version of ovirt-host-deploy is installed?

ovirt-host-deploy-1.6.3-1.el7ev.noarch

Comment 18 Daniel Erez 2017-03-23 08:41:47 UTC
(In reply to Natalie Gavrielov from comment #17)
> (In reply to Daniel Erez from comment #16)
> > @Natalie - which version of ovirt-host-deploy is installed?
> 
> ovirt-host-deploy-1.6.3-1.el7ev.noarch

@Simone - how can I check if the commit is included in this version?

Comment 19 Daniel Erez 2017-03-23 08:58:43 UTC
According to the daemon log [1], it's seems that there wasn't a certificate issue, but a different network error - resolved as part of bug 1430243. 
A workaround to this issue is to manually increase the value of UploadImageXhrTimeoutInSeconds config value in the vdc_options table (e.g. from 10 to 120).

@Natalie - can you please try that?


[1]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__
    resp = self.dispatch(request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch
    return method(*match.groups())
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py", line 175, in put
    op.run()
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 138, in run
    self._receive_chunk(dst, buf, count)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 174, in _receive_chunk
    raise errors.PartialContent(self.size, self.done)
PartialContent: Requested 8388608 bytes, available 1433600 bytes

Comment 20 Natalie Gavrielov 2017-03-23 09:31:16 UTC
(In reply to Daniel Erez from comment #19)
> According to the daemon log [1], it's seems that there wasn't a certificate
> issue, but a different network error - resolved as part of bug 1430243. 
> A workaround to this issue is to manually increase the value of
> UploadImageXhrTimeoutInSeconds config value in the vdc_options table (e.g.
> from 10 to 120).
> 
> @Natalie - can you please try that?
> 
> 
> [1]
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line
> 48, in __call__
>     resp = self.dispatch(request)
>   File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line
> 73, in dispatch
>     return method(*match.groups())
>   File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py",
> line 175, in put
>     op.run()
>   File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py",
> line 138, in run
>     self._receive_chunk(dst, buf, count)
>   File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py",
> line 174, in _receive_chunk
>     raise errors.PartialContent(self.size, self.done)
> PartialContent: Requested 8388608 bytes, available 1433600 bytes

This looks like a different issue, see the timeline in the log, the restart I performed to the daemon was at: 2017-03-22 17:49:54,151
These traceback errors your referring to happened later on (when upload actually started working).

And if I recall correctly, disk size that was set in the UI was 8gb but the actual data that was transferred is around 1.5gb - so it might be just that (probably a bug.. but unrelated to this topic). 

If you still need this workaround to be tested It's best if this needinfo will move to its relevant bug.

Comment 21 Daniel Erez 2017-03-23 09:48:26 UTC
(In reply to Natalie Gavrielov from comment #20)
> (In reply to Daniel Erez from comment #19)
> > According to the daemon log [1], it's seems that there wasn't a certificate
> > issue, but a different network error - resolved as part of bug 1430243. 
> > A workaround to this issue is to manually increase the value of
> > UploadImageXhrTimeoutInSeconds config value in the vdc_options table (e.g.
> > from 10 to 120).
> > 
> > @Natalie - can you please try that?
> > 
> > 
> > [1]
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line
> > 48, in __call__
> >     resp = self.dispatch(request)
> >   File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line
> > 73, in dispatch
> >     return method(*match.groups())
> >   File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py",
> > line 175, in put
> >     op.run()
> >   File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py",
> > line 138, in run
> >     self._receive_chunk(dst, buf, count)
> >   File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py",
> > line 174, in _receive_chunk
> >     raise errors.PartialContent(self.size, self.done)
> > PartialContent: Requested 8388608 bytes, available 1433600 bytes
> 
> This looks like a different issue, see the timeline in the log, the restart
> I performed to the daemon was at: 2017-03-22 17:49:54,151
> These traceback errors your referring to happened later on (when upload
> actually started working).
> 
> And if I recall correctly, disk size that was set in the UI was 8gb but the
> actual data that was transferred is around 1.5gb - so it might be just that
> (probably a bug.. but unrelated to this topic). 
> 
> If you still need this workaround to be tested It's best if this needinfo
> will move to its relevant bug.

I still don't se an SSL error.. Can you please attach the proxy log?

Comment 22 Daniel Erez 2017-03-23 10:13:10 UTC
(In reply to Natalie Gavrielov from comment #13)
> Scenario performed:
> -------------------
> 
> 1. On a clean RHV hypervisor, performed hosted engine installation.
> 2. Configured certificates for both Chrome and Firefox, with all required
> (and 
>    possible) permissions. 
> 3. Tried to upload an image via GUI (disks tab -> upload)
>    * Tried both RAW and Qcow2 v3 formats.
> 
> Result:
> -------
> 
> All attempts to upload failed (switching status to "Paused by the system").
> 
> 
> Important notes:
> ----------------
>  
> 1. After manually restarting the ovirt-imageio-daemon upload image works!
> 2. I looked for the restart of the ovirt-imageio-daemon, that host deploy
> should do in ovirt-imageio-daemon.log and couldn't find any evidence for
> this restart (only the one I performed manually).
> 3. ovirt-imageio-proxy was not installed (never worked with hosted engine
> before but I would guess it's not supposed to be installed..)

ovirt-imageio-proxy should be installed during ovirt-hosted-engine-setup wizard. Have you installed the proxy manually?

> 
> Builds used:
> RHVM 4.1.1.2-0.1.el7
> ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
> 
> Will attach logs in no time.

Comment 23 Natalie Gavrielov 2017-03-23 10:25:20 UTC
Created attachment 1265673 [details]
image proxy, engine, server

(In reply to Daniel Erez from comment #21) 
> I still don't se an SSL error.. Can you please attach the proxy log?

Attached.

(In reply to Daniel Erez from comment #22)
> ovirt-imageio-proxy should be installed during ovirt-hosted-engine-setup > wizard. Have you installed the proxy manually?

It was installed automatically - I missed it because the engine is nested.. and I'm not used it :\

Comment 24 Daniel Erez 2017-03-23 10:33:45 UTC
(In reply to Natalie Gavrielov from comment #23)
> Created attachment 1265673 [details]
> image proxy, engine, server
> 
> (In reply to Daniel Erez from comment #21) 
> > I still don't se an SSL error.. Can you please attach the proxy log?
> 
> Attached.
> 
> (In reply to Daniel Erez from comment #22)
> > ovirt-imageio-proxy should be installed during ovirt-hosted-engine-setup > wizard. Have you installed the proxy manually?
> 
> It was installed automatically - I missed it because the engine is nested..
> and I'm not used it :\

(So did you manually reinstalled the proxy then?)
What are the exact installation steps of hosted engine?

Comment 25 Daniel Erez 2017-03-23 10:50:14 UTC
Also, please attach the ovirt-host-deploy logs..

Comment 26 Natalie Gavrielov 2017-03-23 11:15:50 UTC
Created attachment 1265686 [details]
host-deploy logs

(In reply to Daniel Erez from comment #25)
> Also, please attach the ovirt-host-deploy logs..

Attached.

(In reply to Daniel Erez from comment #24)
> (In reply to Natalie Gavrielov from comment #23)
> > Created attachment 1265673 [details]
> > image proxy, engine, server
> > 
> > (In reply to Daniel Erez from comment #21) 
> > > I still don't se an SSL error.. Can you please attach the proxy log?
> > 
> > Attached.
> > 
> > (In reply to Daniel Erez from comment #22)
> > > ovirt-imageio-proxy should be installed during ovirt-hosted-engine-setup > wizard. Have you installed the proxy manually?
> > 
> > It was installed automatically - I missed it because the engine is nested..
> > and I'm not used it :\
> 
> (So did you manually reinstalled the proxy then?)
> What are the exact installation steps of hosted engine?

No, I did not install / reinstall the proxy - it was installed by the hosted engine installation.

I used a GUI installation provided by the Cockpit and rhevm-appliance, the flow is next-next-next-wait-done similar to this flow:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/html/self-hosted_engine_guide/deploying_self-hosted_engine_on_rhvh .

more about Cockpit:
https://bugzilla.redhat.com/show_bug.cgi?id=1190758
http://cockpit-project.org/

Comment 29 Daniel Erez 2017-03-23 18:39:09 UTC
@Sandro - is a new rhvh build planned for 4.1.1 (which includes ovirt-host-deploy-1.6.3)?

Comment 30 Simone Tiraboschi 2017-03-24 16:43:12 UTC
(In reply to Daniel Erez from comment #18)
> @Simone - how can I check if the commit is included in this version?

https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-host-deploy+branch:ovirt-host-deploy-1.6

It should be there since ovirt-host-deploy-1.6.1

Comment 31 Tal Nisan 2017-03-27 16:21:45 UTC
This should be fixed in 4.1.1-1, moving to MODIFIED

Comment 32 Sandro Bonazzola 2017-03-28 11:28:50 UTC
If it was in 1.6.1, it's already in last week build.

Comment 33 Nikolai Sednev 2017-04-03 10:01:21 UTC
I've deployed a clean HE environment last week with Red Hat Virtualization Manager Version: 4.1.1.6 and then upgraded to 4.1.1.7-0.1.el7, then tried to upload two qcow2 images of rhel-guest-image, I've got the same result as described in https://bugzilla.redhat.com/show_bug.cgi?id=1411491#c13. 
ovirt-imageio-daemon.service was running on both my hosted-engine-hosts.

Following https://bugzilla.redhat.com/show_bug.cgi?id=1411491#c22, ovirt-imageio-proxy is not automatically installed on hoststed-engine-hosts, not in 3.5, not in 3.6.11, not in 4.0.x, neither in latest 4.1.1.7-0.1.el7,so I had to install them manually.


I've manually restarted ovirt-imageio-daemon on both hosts and retried image upload, for this time it wass successful.

Can you check if ovirt-imageio-daemon should automatically get installed during ovirt-hosted-engine-setup please?

Comment 34 Nikolai Sednev 2017-04-04 14:36:22 UTC
1)Deployed fresh and clean hosted-engine over NFS and added two data storage domains over NFS.
2)Added certificate http://<engine_url>/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA using Firefox 52.0 (64-bit).
3)Tried to upload two qcow2 images and failed with "Paused by system" in UI.

Components on engine:
ovirt-imageio-proxy-setup-1.0.0-0.el7ev.noarch
ovirt-imageio-proxy-1.0.0-0.el7ev.noarch
rhev-guest-tools-iso-4.1-5.el7ev.noarch
rhevm-branding-rhev-4.1.0-1.el7ev.noarch
rhevm-4.1.1.7-0.1.el7.noarch
rhevm-doc-4.1.0-3.el7ev.noarch
rhevm-setup-plugins-4.1.1-1.el7ev.noarch
rhevm-dependencies-4.1.1-1.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild@x86-030.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Components on hosts:
libvirt-client-2.0.0-10.el7_3.5.x86_64
ovirt-hosted-engine-setup-2.1.0.5-1.el7ev.noarch
ovirt-host-deploy-1.6.3-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
mom-0.5.9-1.el7ev.noarch
vdsm-4.19.10.1-1.el7ev.x86_64
ovirt-hosted-engine-ha-2.1.0.5-1.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
Linux version 3.10.0-514.16.1.el7.x86_64 (mockbuild@x86-039.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Fri Mar 10 13:12:32 EST 2017
Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Fri Mar 10 13:12:32 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

As this is a clean and fresh deployment on pair of fresh hosts, I expect from the system to work out of the box.
Please consider moving this bug back to assigned.

Comment 35 Nikolai Sednev 2017-04-04 14:38:40 UTC
Created attachment 1268681 [details]
Screenshot from 2017-04-04 17-38-20.png

Comment 36 Nikolai Sednev 2017-04-04 14:44:37 UTC
Created attachment 1268701 [details]
sosreport-nsednev-he-4.scl.lab.tlv.redhat.com-20170404173939.tar.xz

Comment 37 Nikolai Sednev 2017-04-04 14:45:14 UTC
Created attachment 1268702 [details]
sosreport-puma18.scl.lab.tlv.redhat.com-20170404173912.tar.xz

Comment 38 Nikolai Sednev 2017-04-04 14:46:00 UTC
Created attachment 1268703 [details]
sosreport-puma18.scl.lab.tlv.redhat.com-20170404173912.tar.xz

Comment 39 Nikolai Sednev 2017-04-04 14:49:28 UTC
Created attachment 1268705 [details]
sosreport-puma19.scl.lab.tlv.redhat.com-20170404173929.tar.xz

Comment 40 Nikolai Sednev 2017-04-04 15:53:42 UTC
ovirt-host-deploy-1.6.3-1.el7ev.noarch is also installed on both hosts and engine:
puma18 ~]# rpm -qa | grep host-deploy
ovirt-host-deploy-1.6.3-1.el7ev.noarch

nsednev-he-4 ~]#  rpm -qa | grep host-deploy
ovirt-host-deploy-java-1.6.3-1.el7ev.noarch
ovirt-host-deploy-1.6.3-1.el7ev.noarch

Comment 41 Natalie Gavrielov 2017-04-04 15:56:21 UTC
Following Nikolai's comment 34 and comment 40, reopening this issue.

Comment 42 Red Hat Bugzilla Rules Engine 2017-04-04 15:56:30 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 43 Daniel Erez 2017-05-07 09:00:03 UTC
Hi Nikolai,

Can you please attach host-deploy/engine-setup/engine/imageio-proxy/imageio-daemon logs?

Comment 44 Nikolai Sednev 2017-05-07 10:30:15 UTC
Can you please address the comment #43?

Comment 45 Yaniv Lavi 2017-06-07 11:06:33 UTC
Please reopen if you can provide the needed logs.

Comment 46 Natalie Gavrielov 2017-06-07 11:29:28 UTC
Hi All,

I wasn't around the past week, so was unable to address the issue.
Anyway, since the environment is not mine, I don't have these logs.

Yaniv Levi,

I don't really understand why closing this bug - this issue was fixed but not yet verified, IMHO it cannot be closed as insufficient data, it's best to move this to ON_QA.

Comment 47 Nikolai Sednev 2017-06-08 14:27:45 UTC
1. On a clean RHEL7.3 pair of hosts, deployed hosted engine.
2. Configured certificates for both Chrome and Firefox, with all required (and 
   possible) permissions. 
3. Tried to upload an image via GUI (disks tab -> upload)
   * Tried both RAW and Qcow2 v3 formats.

Result:
-------

All attempts to upload passed.

I did not tested this on NGN (RHVH hosts), so I can't provide any information on how this is working there.

Engine:
nsednev-he-1 ~]#  rpm -qa | grep host-deploy
ovirt-host-deploy-1.6.5-1.el7ev.noarch
ovirt-host-deploy-java-1.6.5-1.el7ev.noarch

rhev-guest-tools-iso-4.1-5.el7ev.noarch
rhevm-dependencies-4.1.1-1.el7ev.noarch
rhevm-4.1.3.1-0.1.el7.noarch
rhevm-setup-plugins-4.1.2-1.el7ev.noarch
rhevm-branding-rhev-4.1.0-1.el7ev.noarch
rhevm-doc-4.1.3-1.el7ev.noarch
Linux version 3.10.0-514.16.1.el7.x86_64 (mockbuild@x86-039.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Fri Mar 10 13:12:32 EST 2017
Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Fri Mar 10 13:12:32 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)


Hosts:
alma04 ~]# rpm -qa | grep host-deploy
ovirt-host-deploy-1.6.5-1.el7ev.noarch

ovirt-vmconsole-1.0.4-1.el7ev.noarch
vdsm-4.19.17-1.el7ev.x86_64
ovirt-host-deploy-1.6.5-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
mom-0.5.9-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.10.x86_64
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-hosted-engine-setup-2.1.2-2.el7ev.noarch
ovirt-setup-lib-1.1.1-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.9.x86_64
ovirt-hosted-engine-ha-2.1.1-1.el7ev.noarch
Linux version 3.10.0-514.21.1.el7.x86_64 (mockbuild@x86-020.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Apr 22 02:41:35 EDT 2017
Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Comment 48 Nikolai Sednev 2017-06-11 13:59:25 UTC
Regarding HE deployment on RHVH4.1, there are some problems at this moment which prevents me from trying reproduction on NGN hosts.
Please see https://bugzilla.redhat.com/show_bug.cgi?id=1457684#c12 for more details.

Comment 49 Nikolai Sednev 2017-06-19 12:25:02 UTC
Reproduced on RHVH with these components:
sanlock-3.4.0-1.el7.x86_64
ovirt-node-ng-nodectl-4.1.3-0.20170608.1.el7.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
mom-0.5.9-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.10.x86_64
vdsm-4.19.18-1.el7ev.x86_64
ovirt-host-deploy-1.6.6-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.9.x86_64
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-hosted-engine-ha-2.1.2-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.3-1.el7ev.noarch
ovirt-setup-lib-1.1.3-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
Linux version 3.10.0-514.21.1.el7.x86_64 (mockbuild@x86-020.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Apr 22 02:41:35 EDT 2017
Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 7.3

Engine:
rhevm-dependencies-4.1.1-1.el7ev.noarch
rhevm-doc-4.1.3-1.el7ev.noarch
rhevm-4.1.3.4-0.1.el7.noarch
rhev-guest-tools-iso-4.1-5.el7ev.noarch
rhevm-setup-plugins-4.1.2-1.el7ev.noarch
rhevm-branding-rhev-4.1.0-2.el7ev.noarch
Linux version 3.10.0-514.21.1.el7.x86_64 (mockbuild@x86-020.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Apr 22 02:41:35 EDT 2017
Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.4 Beta (Maipo)

Logs from the engine and the host being attached.
Please also see the screencast as being attached.
Moving back to assigned.

Comment 50 Red Hat Bugzilla Rules Engine 2017-06-19 12:25:09 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 51 Nikolai Sednev 2017-06-19 12:25:37 UTC
Created attachment 1289077 [details]
screencast

Comment 52 Nikolai Sednev 2017-06-19 12:28:53 UTC
Created attachment 1289078 [details]
sosreport from the engine

Comment 53 Nikolai Sednev 2017-06-19 12:29:50 UTC
Created attachment 1289080 [details]
sosreport from puma18 the host

Comment 54 Nikolai Sednev 2017-06-19 12:46:14 UTC
(In reply to Daniel Erez from comment #43)
> Hi Nikolai,
> 
> Can you please attach
> host-deploy/engine-setup/engine/imageio-proxy/imageio-daemon logs?
Just attaching the required log here, it is also available in sosreport from puma18:
/var/log/ovirt-imageio-daemon/daemon.log

Comment 55 Nikolai Sednev 2017-06-19 12:46:43 UTC
Created attachment 1289096 [details]
ovirt-imageio-daemon daemon.log

Comment 56 Daniel Erez 2017-06-19 13:10:31 UTC
@Nikolai - can you please also attach engine-setup/imageio-proxy/imageio-daemon?
Not sure it's the same issue, is restarting the the imageio-daemon solves the issue?

Comment 57 Nikolai Sednev 2017-06-19 13:45:06 UTC
(In reply to Daniel Erez from comment #56)
> @Nikolai - can you please also attach
> engine-setup/imageio-proxy/imageio-daemon?
> Not sure it's the same issue, is restarting the the imageio-daemon solves
> the issue?

Yes, it solves, all logs were already attached within the sosreport.

Comment 58 Daniel Erez 2017-06-19 14:21:39 UTC
@Nikolai - Can you please also upload image-proxy.log
It should be located at /var/log/ovirt-imageio-proxy

Comment 59 Nikolai Sednev 2017-06-19 14:52:20 UTC
Created attachment 1289119 [details]
imageproxy log from an engine

Comment 60 Nikolai Sednev 2017-06-19 14:53:03 UTC
Created attachment 1289120 [details]
image-proxy.log.2017-06-18 from an engine

Comment 61 Daniel Erez 2017-06-19 14:59:56 UTC
(In reply to Nikolai Sednev from comment #60)
> Created attachment 1289120 [details]
> image-proxy.log.2017-06-18 from an engine

The proxy seems empty. Are you connection to a different proxy perhaps? Is uploading currently works?

Comment 62 Nikolai Sednev 2017-06-19 15:19:17 UTC
(In reply to Daniel Erez from comment #61)
> (In reply to Nikolai Sednev from comment #60)
> > Created attachment 1289120 [details]
> > image-proxy.log.2017-06-18 from an engine
> 
> The proxy seems empty. Are you connection to a different proxy perhaps? Is
> uploading currently works?

Uploading currently fails with the same problem as before.
Log really looks strange on engine:
nsednev-he-4 ~]# cat /var/log/ovirt-imageio-proxy/image-proxy.log
(MainThread) INFO 2017-06-19 07:44:58,282 image_proxy:35:root:(main) Server shut down, exiting
(MainThread) INFO 2017-06-19 07:46:46,650 image_proxy:26:root:(main) Server started, successfully notified systemd
(MainThread) INFO 2017-06-19 11:02:39,873 image_proxy:26:root:(main) Server started, successfully notified systemd


Log on host puma18 looks as follows:

puma18 ~]# cat /var/log/ovirt-imageio-daemon/daemon.log 
2017-06-19 13:38:36,021 INFO    (MainThread) [server] Starting (version 1.0.0)
2017-06-19 13:38:36,026 INFO    (MainThread) [server] Ready for requests
2017-06-19 15:02:35,941 INFO    (MainThread) [server] Received signal 15, shutting down
2017-06-19 15:02:37,655 INFO    (MainThread) [server] Stopped
2017-06-19 15:02:37,905 INFO    (MainThread) [server] Starting (version 1.0.0)
2017-06-19 15:02:37,996 INFO    (MainThread) [server] Ready for requests
2017-06-19 15:21:46,410 INFO    (ticket.server) [tickets] Adding ticket {u'uuid': u'56fc6346-acf4-497b-a61c-7cbff186bd20', u'ops': [u'write'], u'url': ParseResult(scheme=u'file', netloc=u'', path=u'/rhev/data-center/00000001-0001-0001-0001-000000000311/0b4c1b5c-7421-447b-911a-735af600c453/images/08e5a6e7-ed12-4b38-b0ce-2f029a45e29f/171b400a-9a1e-4f26-a3e3-a88872f0793b', params='', query='', fragment=''), 'expires': 4301179, u'timeout': 300, u'size': 421527552}
2017-06-19 15:21:46,411 INFO    (ticket.server) [web] / - PUT /56fc6346-acf4-497b-a61c-7cbff186bd20 200 0 (0.00s)
2017-06-19 15:22:01,342 INFO    (ticket.server) [tickets] Removing ticket 56fc6346-acf4-497b-a61c-7cbff186bd20
2017-06-19 15:22:01,342 INFO    (ticket.server) [web] / - DELETE /56fc6346-acf4-497b-a61c-7cbff186bd20 204 0 (0.00s)
2017-06-19 18:16:13,374 INFO    (ticket.server) [tickets] Adding ticket {u'uuid': u'eecb87d1-582d-4b2a-a46e-da59049e06fd', u'ops': [u'write'], u'url': ParseResult(scheme=u'file', netloc=u'', path=u'/rhev/data-center/00000001-0001-0001-0001-000000000311/0b4c1b5c-7421-447b-911a-735af600c453/images/a119b118-0b7d-4b25-87ec-caeaa36323f2/1284bf0f-bd51-43e4-83ef-138dbb271962', params='', query='', fragment=''), 'expires': 4311646, u'timeout': 300, u'size': 421527552}
2017-06-19 18:16:13,374 INFO    (ticket.server) [web] / - PUT /eecb87d1-582d-4b2a-a46e-da59049e06fd 200 0 (0.00s)
2017-06-19 18:16:27,581 INFO    (ticket.server) [tickets] Removing ticket eecb87d1-582d-4b2a-a46e-da59049e06fd
2017-06-19 18:16:27,582 INFO    (ticket.server) [web] / - DELETE /eecb87d1-582d-4b2a-a46e-da59049e06fd 204 0 (0.00s)

Comment 63 Nikolai Sednev 2017-06-19 15:19:59 UTC
Created attachment 1289138 [details]
screencast

Comment 64 Daniel Erez 2017-06-19 15:54:10 UTC
I've tried to upload an image using you env with FireFox 53 and it seems to work fine. Which browser version are you using? Did you restart the browser after importing the certificate? Can you please try to upload from another machine (With latest FireFox) in order to understand whether it's a browser issue indeed.

Comment 65 Nikolai Sednev 2017-06-20 09:38:54 UTC
I was using:

Firefox 52.1.0 (64-bit).
I did restarted browser, tried to clean also cookies and history, as well as restarting my laptop. Nothing helped, upload being paused.

Same thing happened also for Google Chrome.
Version 59.0.3071.86 (Official Build) (64-bit)


Comment 66 Nikolai Sednev 2017-06-20 13:13:33 UTC
Returning to ON_QA as I've discovered that in my case somehow the issue was caused by stale CA certificate. Although I've installed a new and fresh CA certificate for new environment, somehow an old CA certificate was used instead of new during image upload and was failing to get uploaded. Once I've manually removed both old and new CA certificates and then manually re-added CA certificate, then upload succeeded.

Comment 67 Nikolai Sednev 2017-06-21 11:05:46 UTC
Works for me on cleanly deployed HE environment over NFS, on pair of RHVH4.1(based on el7.3) aka NGN hosts, with two NFS data storage domains.

Components on hosts:
sanlock-3.4.0-1.el7.x86_64
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-hosted-engine-ha-2.1.0.6-1.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-node-ng-nodectl-4.1.1-0.20170404.0.el7.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
mom-0.5.9-1.el7ev.noarch
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-hosted-engine-setup-2.1.0.6-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64
vdsm-4.19.15-1.el7ev.x86_64
ovirt-host-deploy-1.6.5-1.el7ev.noarch
Linux version 3.10.0-514.21.2.el7.x86_64 (mockbuild@x86-037.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sun May 28 17:08:21 EDT 2017
Linux 3.10.0-514.21.2.el7.x86_64 #1 SMP Sun May 28 17:08:21 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 7.3

Components on Engine:
rhevm-dependencies-4.1.1-1.el7ev.noarch
rhevm-doc-4.1.3-1.el7ev.noarch
rhevm-4.1.3.4-0.1.el7.noarch
rhev-guest-tools-iso-4.1-5.el7ev.noarch
rhevm-setup-plugins-4.1.2-1.el7ev.noarch
rhevm-branding-rhev-4.1.0-2.el7ev.noarch
Linux version 3.10.0-514.21.1.el7.x86_64 (mockbuild@x86-020.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Apr 22 02:41:35 EDT 2017
Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.4 Beta (Maipo)


I've deleted an old CA certificate of the engine and then got a new after deployed a fresh HE environment.
Uploaded ISO image successfully.
Moving this bug to verified as it works for me on RHVH4.1(based on el7.3) and also on RHEL7.3.

Comment 68 Nikolai Sednev 2017-06-21 11:18:22 UTC
Attaching a successfully upload screencast here:
https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88cFpnc0N0d0NrVlk/view?usp=sharing

Comment 69 Nikolai Sednev 2017-06-21 11:25:39 UTC
Created attachment 1290023 [details]
Screenshot from 2017-06-21 14-24-42.png


Note You need to log in before you can comment on or make changes to this bug.