Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510398 - Unable to upload glance image
Summary: Unable to upload glance image
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-swift
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 10.0 (Newton)
Assignee: Pete Zaitcev
QA Contact: Mike Abrams
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-07 10:17 UTC by Chaitanya
Modified: 2018-05-16 12:21 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-05 09:39:42 UTC
tshefi: automate_bug-


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3447011 None None None 2018-05-16 12:21:55 UTC

Description Chaitanya 2017-11-07 10:17:42 UTC
Description of problem:
Unable to upload glance image. Glance backend is swift:

~~~
[glance_store]
stores = glance.store.http.Store,glance.store.swift.Store
default_store = swift
swift_store_endpoint_type = internalURL
swift_store_create_container_on_put = True
default_swift_reference = ref1
swift_store_config_file = /etc/glance/glance-swift.conf
~~~

While uploading the image, following error is seen:

~~~
$ openstack image create --container-format bare --disk-format qcow2 --file virt-cirros-0.3.4-x86_64-disk.img cirros
500 Internal Server Error
The server has either erred or is incapable of performing the requested operation.
    (HTTP 500)
~~~

Checking the glance api logs:

~~~
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi [req-2a64f4f9-600e-43ee-ba6c-eca90cc6f07b 1b895804e0454211a027a57397ff2b38 db35f0d1c355419c
bd3f52d16391667e - default default] Caught error: Failed to add object to Swift.
Got error from Swift: put_object('glance', 'b97d4a3d-d01b-4f89-b946-b7bc8f382040-00001', ...) failure and no ability to reset contents for reuploa
d..
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi Traceback (most recent call last):
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1113, in __call__
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     request, **action_args)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1152, in dispatch
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     return method(*args, **kwargs)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 363, in wrapped
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 250, in upload
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     self._restore(image_repo, image)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     self.force_reraise()
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_rerais
e
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     image.set_data(data, size)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     self.base.set_data(data, size)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     self.force_reraise()
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     self.repo.set_data(data, size)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 185, in set_data
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     self.image.set_data(data, size=size)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     verifier=verifier)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 452, in add_to_backend
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     verifier)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 425, in store_add_to_backend
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     verifier=verifier)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/swift/store.py", line 966, in add
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi     raise glance_store.BackendException(msg)
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi BackendException: Failed to add object to Swift.
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi Got error from Swift: put_object('glance', 'b97d4a3d-d01b-4f89-b946-b7bc8f382040-00001', ...) failure and no ability to reset contents for reupload..
2017-11-07 16:47:17.425 88158 ERROR glance.common.wsgi
~~~

In swift.log file following errors are seen (related to image b97d4a3d-d01b-4f89-b946-b7bc8f382040-00001):

~~~
Nov  7 11:47:16 localhost object-server: ERROR __call__ error with PUT /d1/968/AUTH_bf0fd962da584525861d1ae195537d85/glance/b97d4a3d-d01b-4f89-b946-b7bc8f382040-00001 : #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/site-packages/swift/obj/server.py", line 1102, in __call__#012    res = getattr(self, req.method)(req)#012  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 1588, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.7/site-packages/swift/obj/server.py", line 762, in PUT#012    for chunk in iter(timeout_reader, ''):#012  File "/usr/lib/python2.7/site-packages/swift/obj/server.py", line 410, in timeout_reader#012    return file_like.read(self.network_chunk_size)#012  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 194, in read#012    return self._chunked_read(self.rfile, length)#012  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 184, in _chunked_read#012    raise ChunkReadError(err)#012ChunkReadError: invalid literal for int() with base 16: '' (txn: tx8bce20f50b594e5f82ad1-005a01e393)
Nov  7 11:47:16 localhost object-server: 192.0.2.8 - - [07/Nov/2017:16:47:16 +0000] "PUT /d1/968/AUTH_bf0fd962da584525861d1ae195537d85/glance/b97d4a3d-d01b-4f89-b946-b7bc8f382040-00001" 500 863 "PUT http://192.0.2.13:8080/v1/AUTH_bf0fd962da584525861d1ae195537d85/glance/b97d4a3d-d01b-4f89-b946-b7bc8f382040-00001" "tx8bce20f50b594e5f82ad1-005a01e393" "proxy-server 287935" 0.5049 "-" 288233 0
~~~

Using the latest swift packages:

~~~
# rpm -qa | grep -i swift
openstack-swift-account-2.10.2-2.el7ost.noarch
python-swiftclient-3.1.0-1.el7ost.noarch
openstack-swift-proxy-2.10.2-2.el7ost.noarch
python-swift-2.10.2-2.el7ost.noarch
openstack-swift-container-2.10.2-2.el7ost.noarch
openstack-swift-plugin-swift3-1.11.0-7.el7ost.noarch
puppet-swift-9.5.0-1.el7ost.noarch
openstack-swift-object-2.10.2-2.el7ost.noarch
~~~

Version-Release number of selected component (if applicable):
RHOSP 10

How reproducible:
Always

Comment 32 o.chaze 2018-03-16 13:11:30 UTC
Hi Chaitanya,

Having the same issue here, did you find out what was the issue by any chance ?

Thx.

Comment 33 Chaitanya 2018-03-16 18:33:14 UTC
Hi,
  
> Hi Chaitanya,
> 
> Having the same issue here, did you find out what was the issue by any
> chance ?

The issue was that the jumbo frames were enabled on the storage management network but corresponding setting was not made at switch level. Once this was corrected, image upload to glance worked correctly.

Hope this helps.
> 
> Thx.

Comment 34 o.chaze 2018-03-19 12:11:48 UTC
It helped ! Tthat was the problem, thanks :)

Comment 37 skh251 2018-05-06 20:04:29 UTC
it seems i have same issue on swift, I did not change default value of chunks size, does default value suppose jumbo frame? Would please describe your solution, i can configure our switch to support jumbo frame.


object-server - - -  ERROR __call__ error with PUT /sdb/20637/AUTH_4a3e3bbd118c41b48238766e82ca5e4a/muc1/a17d85a56fc36a13e16ada9a492e7f98d214bd36.jpg : #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 1032, in __call__#012    res = method(req)#012  File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1450, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 704, in PUT#012    for chunk in iter(timeout_reader, ''):#012  File "/usr/lib/python2.7/dist-packages/swift/obj/server.py", line 399, in timeout_reader#012    return file_like.read(self.network_chunk_size)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 194, in read#012    return self._chunked_read(self.rfile, length)#012  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 184, in _chunked_read#012    raise ChunkReadError(err)#012ChunkReadError: invalid literal for int() with base 16: '' (txn: txae003dbdb32743c694c29-005aef5a4f)

Comment 38 skh251 2018-05-08 06:12:59 UTC
I set mtu 9000 on my switch  and our chunk size as default 65536. would please help

Comment 39 o.chaze 2018-05-08 07:26:34 UTC
(In reply to skh251 from comment #38)
> I set mtu 9000 on my switch  and our chunk size as default 65536. would
> please help

Hi,

The problem for me was on the swift-proxy server side, running behind a haproxy. The mtu was 1500 while the swift clients and storage nodes were in 9000, thus breaking the whole chain.
In order to identify the bad guy I ran a ping against each server with a packet size > 1500. 
ie: ping -s 8000 $hostname

Hope it helps


Note You need to log in before you can comment on or make changes to this bug.