Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1357548 - RFE: upload image - verify image format on client side
Summary: RFE: upload image - verify image format on client side
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.0.1.2
Hardware: Unspecified
OS: Unspecified
medium
high vote
Target Milestone: ovirt-4.0.4
: 4.0.4
Assignee: Daniel Erez
QA Contact: Natalie Gavrielov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-18 13:29 UTC by Natalie Gavrielov
Modified: 2016-09-26 12:40 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-26 12:40:02 UTC
oVirt Team: Storage
tnisan: ovirt-4.0.z?
ngavrilo: planning_ack?
rule-engine: devel_ack+
rule-engine: testing_ack+


Attachments (Terms of Use)
engine.log, vdsm.log, image-proxy.log (deleted)
2016-07-18 13:29 UTC, Natalie Gavrielov
no flags Details
libvirtd.log - just in case (deleted)
2016-07-18 14:02 UTC, Natalie Gavrielov
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 61035 master ABANDONED core: volume verification - an event log on failure 2016-07-20 09:05:17 UTC
oVirt gerrit 61179 master MERGED webadmin: image upload - image info detection 2016-08-10 09:59:29 UTC
oVirt gerrit 62426 ovirt-engine-4.0 MERGED webadmin: image upload - image info detection 2016-08-18 12:50:13 UTC
Red Hat Bugzilla 1344285 None CLOSED Fail fast on not-valid QCOW image upload 2019-03-27 13:18:08 UTC

Internal Links: 1344285

Description Natalie Gavrielov 2016-07-18 13:29:48 UTC
Created attachment 1181076 [details]
engine.log, vdsm.log, image-proxy.log

Description of problem:

Uploading image with Image Type raw cause image verification failure.

Version-Release number of selected component:
rhevm-4.0.2-0.2.rc1.el7ev.noarch
ovirt-imageio-common-0.3.0-0.el7ev.noarch
ovirt-imageio-proxy-0.3.0-0.el7ev.noarch
vdsm-4.18.6-1.el7ev.x86_64
ovirt-imageio-daemon-0.3.0-0.el7ev.noarch

How reproducible:
100%

Steps to Reproduce:

Upload image -
1. Image Type: QCOW2
2. Image Type: Raw 

Actual results:

1. Seems that when uploading image type QCOW2 passed successfully - image status when done is "OK".
2. When finishing: status failed then finalizing failure, and then Illegal.
From engine.log:
2016-07-18 16:15:18,764 ERROR [org.ovirt.engine.core.bll.storage.disk.image.UploadDiskImageCommand] (DefaultQuartzScheduler8) [790e9c2a] Failed to verify uploaded image: {}: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to VerifyUntrustedVolumeVDS, error = Image verification failed: "reason=Volume's format specified by QEMU is qcow2, while the format specified in VDSM metadata is raw", code = 484 (Failed with error unexpected and code 16)
	at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114) [bll.jar:]
	at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommand.verifyImage(UploadImageCommand.java:286) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommand.handleFinalizingSuccess(UploadImageCommand.java:266) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommand.executeStateHandler(UploadImageCommand.java:147) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommand.proceedCommandExecution(UploadImageCommand.java:116) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommandCallback.doPolling(UploadImageCommandCallback.java:13) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:113) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) [:1.8.0_101]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101]
	at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77) [scheduler.jar:]
	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51) [scheduler.jar:]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_101]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]

Expected results:
For the upload to work, and in the end display status "OK". 

Additional info:

Comment 1 Allon Mureinik 2016-07-18 13:40:24 UTC
Natalie, I'm afraid I don't understand the flow here.
Are you selecting QCow in the GUI and then feeding it a RAW file? If not, could you clarify the stages you executed?

Thanks!

Comment 2 Natalie Gavrielov 2016-07-18 14:02:59 UTC
Created attachment 1181123 [details]
libvirtd.log - just in case

Comment 3 Natalie Gavrielov 2016-07-18 14:13:30 UTC
(In reply to Allon Mureinik from comment #1)
> Natalie, I'm afraid I don't understand the flow here.
> Are you selecting QCow in the GUI and then feeding it a RAW file? If not,
> could you clarify the stages you executed?
> 
> Thanks!

Steps are: upload a file using UI, select image type - raw.
When the upload is done, there is the following error:
Failed to VerifyUntrustedVolumeVDS, error = Image verification failed: "reason=Volume's format specified by QEMU is qcow2, while the format specified in VDSM metadata is raw

Note, that when performing the same scenario using image type QCOW2 - it finishes with status OK.

Comment 4 Nir Soffer 2016-07-18 16:43:45 UTC
After talking with Natalie, the system seems to behave as it was designed.

The system failed verification after uploading about 2mb of 3gb image, because
the actual image format was qcow, while the selected image format was raw.

Issues that we may need to investigate:

- According to Natalie, it took 2-3 minutes until the upload failed - I would
  expect the failure after several seconds.

- The UI does not give any clue about what happens. I would expect to see this:

1. Click upload
2. Show "verifying image" indeterminate progress (.e.g spinner)
3. If verification failed, show error dialog in the ui
4. When verification finish, show "uploading image" progress (progress bar)

Since this was not specified when we started to work on this, we will have to 
improve this in the next version.

Comment 5 Daniel Erez 2016-07-18 19:18:43 UTC
Hi Natalie,

Are you using NFS or iSCSI? Thin or Preallocated?
Note that unsupported combinations are now blocked from the UI (should be available on upcoming builds).

Comment 6 Natalie Gavrielov 2016-07-20 09:08:09 UTC
(In reply to Nir Soffer from comment #4)
> After talking with Natalie, the system seems to behave as it was designed.
> 
> The system failed verification after uploading about 2mb of 3gb image,
> because
> the actual image format was qcow, while the selected image format was raw.
> 
> Issues that we may need to investigate:
> 
> - According to Natalie, it took 2-3 minutes until the upload failed - I would
>   expect the failure after several seconds.
> 
> - The UI does not give any clue about what happens. I would expect to see
> this:
> 
> 1. Click upload
> 2. Show "verifying image" indeterminate progress (.e.g spinner)
> 3. If verification failed, show error dialog in the ui
> 4. When verification finish, show "uploading image" progress (progress bar)
> 
> Since this was not specified when we started to work on this, we will have
> to 
> improve this in the next version.


I think there might be some misunderstandings here.. 
It all started when I tried to upload a QCOW disk as Raw and saw this fails within a few minutes (2-3) with no explanation - which is why I thought it is a bug, and opened bug 1357269.
Later on, for reasons I can't explain.. behaviour was changed (even before I added the patch Nir sent me for the logging, and NO, I did not change/touch anything in the environment).
How behaviour was changed?
It started uploading QCOW2 files as Raw:
It uploads the WHOLE file.. and then switches it's state to Illegal..
And then I opened THIS bug.   

So right now.. behaviour is wrong - we upload QCOW2 disk as Raw - all the 3GB.. not just 2-3 MB (eventually it fails.. which is good, but why upload the whole thing?!).

Comment 7 Daniel Erez 2016-07-20 09:16:51 UTC
@Natalie - is it relevant only for file domains?
If so, then it's already handled by bug 1353229 . I.e. uploading as QCOW2 on a file domain is not supported and thus disabled.

BTW, the format verification can currently be done only after uploading the entire file, cause we need the file in the domain in order to test it.. So that's by design.

Comment 8 Daniel Erez 2016-07-20 09:18:40 UTC
Another thing, when I've tried to reproduce it I got a proper event message in the webadmin [*], did you get it as well?

[*] "VDSM localhost command failed: Image verification failed: "reason=Volume's format specified by QEMU is raw, while the format specified in VDSM metadata is qcow2""

Comment 9 Natalie Gavrielov 2016-07-20 09:34:48 UTC
(In reply to Daniel Erez from comment #8)
> Another thing, when I've tried to reproduce it I got a proper event message
> in the webadmin [*], did you get it as well?
> 
> [*] "VDSM localhost command failed: Image verification failed:
> "reason=Volume's format specified by QEMU is raw, while the format specified
> in VDSM metadata is qcow2""

Yes, I did.

Comment 10 Natalie Gavrielov 2016-07-20 14:02:57 UTC
(In reply to Daniel Erez from comment #5)
> Hi Natalie,
> 
> Are you using NFS or iSCSI? Thin or Preallocated?
> Note that unsupported combinations are now blocked from the UI (should be
> available on upcoming builds).

and

(In reply to Daniel Erez from comment #7)
> @Natalie - is it relevant only for file domains?


1. QCOW2 image disk, uploaded as Raw image type, preallocated, block
2. QCOW2 image disk, uploaded as Raw image type, thin provision, file
3. QCOW2 image disk, uploaded as Raw image type, preallocated, file

For all these option it uploads the whole file, then switches state to Illegal.

For the remaining option: QCOW2 image disk, uploaded as Raw image type, thin provision, block
there is a message in the UI: Error while executing action: No Message

Comment 11 Nir Soffer 2016-07-24 13:02:31 UTC
(In reply to Natalie Gavrielov from comment #10)
> (In reply to Daniel Erez from comment #5)
> > Hi Natalie,
> > 
> > Are you using NFS or iSCSI? Thin or Preallocated?
> > Note that unsupported combinations are now blocked from the UI (should be
> > available on upcoming builds).
> 
> and
> 
> (In reply to Daniel Erez from comment #7)
> > @Natalie - is it relevant only for file domains?
> 
> 
> 1. QCOW2 image disk, uploaded as Raw image type, preallocated, block
> 2. QCOW2 image disk, uploaded as Raw image type, thin provision, file
> 3. QCOW2 image disk, uploaded as Raw image type, preallocated, file
> 
> For all these option it uploads the whole file, then switches state to
> Illegal.
> 
> For the remaining option: QCOW2 image disk, uploaded as Raw image type, thin
> provision, block
> there is a message in the UI: Error while executing action: No Message

I suspect that what happens here is this:

1. Staring first upload
2. First image verification fails after the first chunk of the image reach storage
3. Upload is paused, but the system remember the uploaded size (8MiB?)
4. Second upload to same image, system resume upload from last uploaded offset
5. Since we do not upload the first chunk, there is no verification of the first chunk
6. Upload finish
7. Last image verification fails, and image is paused again

So the bug is in step 3 - when upload verification fails, the system must set the
uploaded size to zero. Retrying the upload must start from the first chunk.

Amit, can you confirm this?

Comment 12 Amit Aviram 2016-07-24 13:36:15 UTC
(In reply to Nir Soffer from comment #11)
> (In reply to Natalie Gavrielov from comment #10)
> > (In reply to Daniel Erez from comment #5)
> > > Hi Natalie,
> > > 
> > > Are you using NFS or iSCSI? Thin or Preallocated?
> > > Note that unsupported combinations are now blocked from the UI (should be
> > > available on upcoming builds).
> > 
> > and
> > 
> > (In reply to Daniel Erez from comment #7)
> > > @Natalie - is it relevant only for file domains?
> > 
> > 
> > 1. QCOW2 image disk, uploaded as Raw image type, preallocated, block
> > 2. QCOW2 image disk, uploaded as Raw image type, thin provision, file
> > 3. QCOW2 image disk, uploaded as Raw image type, preallocated, file
> > 
> > For all these option it uploads the whole file, then switches state to
> > Illegal.
> > 
> > For the remaining option: QCOW2 image disk, uploaded as Raw image type, thin
> > provision, block
> > there is a message in the UI: Error while executing action: No Message
> 
> I suspect that what happens here is this:
> 
> 1. Staring first upload
> 2. First image verification fails after the first chunk of the image reach
> storage
> 3. Upload is paused, but the system remember the uploaded size (8MiB?)
> 4. Second upload to same image, system resume upload from last uploaded
> offset
> 5. Since we do not upload the first chunk, there is no verification of the
> first chunk
> 6. Upload finish
> 7. Last image verification fails, and image is paused again
> 
> So the bug is in step 3 - when upload verification fails, the system must
> set the
> uploaded size to zero. Retrying the upload must start from the first chunk.
> 
> Amit, can you confirm this?

No, we don't have the initial verification implemented yet, Bug 1344285 should handle that. 

Natalie, if this bug is about making the upload fail faster when the format is incorrect, then we can set it to be a duplicate of Bug 1344285. 

Otherwise we might consider closing the bug, as its description and title describes a desired flow.

Comment 13 Natalie Gavrielov 2016-08-04 12:56:13 UTC
Irrelevant now.. closing.

Comment 14 Daniel Erez 2016-08-04 13:03:48 UTC
Reopening - will be used as an RFE for client side verification.

Comment 15 Amit Aviram 2016-08-17 10:26:43 UTC
Derez, do you remember to backport this patch?

Comment 16 Daniel Erez 2016-08-17 11:15:36 UTC
(In reply to Amit Aviram from comment #15)
> Derez, do you remember to backport this patch?

Not sure we should.

@Allon - is it 4.0 material or should be postponed to 4.1?

Comment 17 Allon Mureinik 2016-08-17 12:47:37 UTC
(In reply to Daniel Erez from comment #16)
> (In reply to Amit Aviram from comment #15)
> > Derez, do you remember to backport this patch?
> 
> Not sure we should.
> 
> @Allon - is it 4.0 material or should be postponed to 4.1?

4.0.4 material, like the target says.
(Assuming we trust this patch, of course)

Comment 18 Natalie Gavrielov 2016-09-07 10:15:46 UTC
Daniel,

I've noticed that format verification on client's side can only tell if the file is qcow2.. all other files are catalogued as raw.. is this true?

Comment 19 Amit Aviram 2016-09-07 10:19:17 UTC
(In reply to Natalie Gavrielov from comment #18)
> Daniel,
> 
> I've noticed that format verification on client's side can only tell if the
> file is qcow2.. all other files are catalogued as raw.. is this true?

Yes, that's ok. RAW type is just a bunch of bits- without any format.

Comment 20 Natalie Gavrielov 2016-09-07 10:38:26 UTC
(In reply to Amit Aviram from comment #19)
> (In reply to Natalie Gavrielov from comment #18)
> > Daniel,
> > 
> > I've noticed that format verification on client's side can only tell if the
> > file is qcow2.. all other files are catalogued as raw.. is this true?
> 
> Yes, that's ok. RAW type is just a bunch of bits- without any format.

Another thing, qcow is recognized as qcow2..

Comment 21 Natalie Gavrielov 2016-09-07 12:35:55 UTC
Verified,
Everything that is qcow2 / qcow2 v3 - recognized as qcow2, with the right compatibility mode.
qcow is also recognized as qcow2 - I will open a separate bug.
Everything else - recognized as raw.


Builds used:
rhevm-4.0.4-0.1.el7ev.noarch
vdsm-4.18.12-1.el7ev.x86_64
ovirt-imageio-proxy-0.3.0-0.el7ev.noarch
ovirt-imageio-common-0.3.0-0.el7ev.noarch
ovirt-imageio-daemon-0.3.0-0.el7ev.noarch


Note You need to log in before you can comment on or make changes to this bug.