Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1513474

Summary: ImageIO- upload via sdk of QCOW image fails after ~1GB with 'IOError: [Errno 28] No space left on device'
Product: [oVirt] ovirt-engine Reporter: Avihai <aefrat>
Component: BLL.StorageAssignee: Daniel Erez <derez>
Status: CLOSED NOTABUG QA Contact: Raz Tamir <ratamir>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: amureini, bugs, derez
Target Milestone: ovirt-4.2.0Keywords: Reopened
Target Release: ---Flags: rule-engine: ovirt-4.2+
amureini: devel_ack-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-15 14:51:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Attachments:
Description Flags
engine , vdsm ,image_proxy,image_demon logs
none
newer_upload.script
none
old_upload_script none

Description Avihai 2017-11-15 13:38:47 UTC
Description of problem:
Upload via new sdk script (https://gerrit.ovirt.org/#/c/83566/3/sdk/examples/upload_disk.py) a 2G qcow disk.
See the script I used attached.

After ~50% (1G) upload fails with IOError: [Errno 28] No space left on device on deamon.log.

The issue does not happen using the UI so this is a script/SDK issue only.

Version-Release number of selected component (if applicable):

Engine:
ovirt-engine-4.2.0-0.0.master.20171114111003.git7aa1b91.el7.centos.noarch

VDSM:
4.20.7-1.gitc9cf1ee

ImageIO:
imageio-daemon-1.2.0-0.201711091209.git67cd384
ovirt-imageio-proxy-1.2.0-0.201711091209.git67cd384

How reproducible:
100%


Steps to Reproduce:
Upload via new sdk script (https://gerrit.ovirt.org/#/c/83566/3/sdk/examples/upload_disk.py) a 2G qcow disk.
See the script I used attached .

Actual results:
After ~50% (1G) upload fails with IOError: [Errno 28] No space left on device on deamon.log. 


Expected results:


Additional info:
deamon.log:
2017-11-15 14:55:24,973 ERROR   (Thread-1) [web] 10.35.162.7 - PUT /2cbc39a7-d9b7-40dc-bc73-8a34770f789f 500 196 (126.39s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__
    resp = self.dispatch(request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch
    return method(*match.groups())
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py", line 169, in put
    op.run()
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 72, in run
    self._run()
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 160, in _run
    self._receive_chunk(dst, buf, count)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 190, in _receive_chunk
    towrite -= util.uninterruptible(dst.write, wbuf)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/util.py", line 19, in uninterruptible
    return func(*args)
IOError: [Errno 28] No space left on device
2017-11-15 14:57:24,055 INFO    (ticket.server) [tickets] Extending ticket 2cbc39a7-d9b7-40dc-bc73-8a34770f789f, new expiration in 4889587

image_proxy.log:
(Thread-2  ) ERROR 2017-11-15 14:55:24,984 images:185:root:(make_imaged_request) Failed communicating with host: A Connection error occurred.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/images.py", line 176, in make_imaged_request
    timeout=timeout, stream=stream)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))
(Thread-2  ) ERROR 2017-11-15 14:55:25,025 web:89:web:(log_response) 10.35.4.39 - PUT  503 208 (126.58s)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__
    resp = self.dispatch(request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch
    return method(*match.groups())
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 88, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/http_helper.py", line 59, in wrapper
    ret = func(self, *args)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/images.py", line 79, in put
    return self.send_data(self.request)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/images.py", line 116, in send_data
    request.method, imaged_url, headers, body, stream)
  File "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/images.py", line 186, in make_imaged_request
    raise exc.HTTPServiceUnavailable(s)
HTTPServiceUnavailable: Failed communicating with host: A Connection error occurred.

VDSM log:
2017-11-15 14:55:18,766+0200 DEBUG (jsonrpc/7) [storage.TaskManager.Task] (Task='bf1d3e8a-7b14-4303-89eb-214eef526b17') finished: {'result': {u'uuid': u'2cbc39a7-d9b7-40dc-bc73-8a34770f789f', u'ops': [u'write'],
 u'url': u'file:///rhev/data-center/mnt/blockSD/a50928e6-bc2e-47f8-abdd-4a5cb0c5ad35/images/2b289d61-918f-48a3-a690-7e31f9b7397e/6532f11b-97cb-4ee9-a00c-17a958ede431', u'expires': 4889341, u'transferred': 1146093568, u'timeout': 180, u'active': True, u'size': 10737418240}} (task:1201)

Comment 1 Avihai 2017-11-15 13:40:09 UTC
Created attachment 1352614 [details]
engine , vdsm ,image_proxy,image_demon logs

Comment 2 Avihai 2017-11-15 13:59:15 UTC
Created attachment 1352616 [details]
newer_upload.script

Comment 3 Avihai 2017-11-15 13:59:59 UTC
Created attachment 1352617 [details]
old_upload_script

Comment 4 Avihai 2017-11-15 14:00:58 UTC
Issue occured only with the new script, with the old one it works ok.

Comment 5 Allon Mureinik 2017-11-15 14:31:18 UTC
The newer script creates a disk the size of 1GB (on line 80: initial_size = 1 * 2**30, line), instead of a 10GB one like the older script.

Fixing that should fix the bug.

Comment 6 Avihai 2017-11-15 14:49:12 UTC
(In reply to Allon Mureinik from comment #5)
> The newer script creates a disk the size of 1GB (on line 80: initial_size =
> 1 * 2**30, line), instead of a 10GB one like the older script.
> 
> Fixing that should fix the bug.

Fixing the script does not mean the bug is not there.

Why shouldn't it work with 1GB initial size?
1GB is the QCOW disk initial size for any QCOW disk,right ?

Comment 7 Allon Mureinik 2017-11-15 14:51:16 UTC
(In reply to Avihai from comment #6)
> (In reply to Allon Mureinik from comment #5)
> > The newer script creates a disk the size of 1GB (on line 80: initial_size =
> > 1 * 2**30, line), instead of a 10GB one like the older script.
> > 
> > Fixing that should fix the bug.
> 
> Fixing the script does not mean the bug is not there.
> 
> Why shouldn't it work with 1GB initial size?
> 1GB is the QCOW disk initial size for any QCOW disk,right ?

You HAVE to provide the size when creating a disk to upload a file to. Disks can only grow dynamically when a VM is running them, not by any other means.

Comment 8 Avihai 2017-11-15 14:58:28 UTC
This is Nir S. script from which Derez & Nir asked me to use from https://gerrit.ovirt.org/#/c/83566/3/sdk/examples/upload_disk.py 
&
https://gerrit.ovirt.org/gitweb?p=ovirt-engine-sdk.git;a=blob_plain;f=sdk/examples/upload_disk.py;hb=refs/changes/14/83714/3

So I assumed it was correct.

So script is wrong .

Comment 9 Allon Mureinik 2017-11-15 15:06:29 UTC
(In reply to Avihai from comment #8)
> This is Nir S. script from which Derez & Nir asked me to use from
> https://gerrit.ovirt.org/#/c/83566/3/sdk/examples/upload_disk.py 
> &
> https://gerrit.ovirt.org/gitweb?p=ovirt-engine-sdk.git;a=blob_plain;f=sdk/
> examples/upload_disk.py;hb=refs/changes/14/83714/3
> 
> So I assumed it was correct.
> 
> So script is wrong .

To quote the script:
# 2. The disk size is indicated using the 'provisioned_size' attribute,
#    but due to current limitations in the engine, the 'initial_size'
#    attribute also needs to be explicitly provided for _copy on write_
#    disks created on block storage domains, so that all the required
#    space is allocated upfront, otherwise the upload will eventually
#    fail.

But yeah, I guess we could (should?) increase the initial size there.