Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1692437 - [RHOSP10]cinder fails to display volume types, QoS specs and their associations after specifying QoS spec for Ceph backend
Summary: [RHOSP10]cinder fails to display volume types, QoS specs and their associatio...
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 10.0 (Newton)
Hardware: All
OS: All
medium
high
Target Milestone: ---
: 10.0 (Newton)
Assignee: Pablo Caruana
QA Contact: Tzach Shefi
Tana
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-25 15:37 UTC by Alex Stupnikov
Modified: 2019-04-03 04:34 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1626704 None None None 2019-03-29 14:57:17 UTC
OpenStack gerrit 375077 None None None 2019-03-29 14:58:09 UTC

Description Alex Stupnikov 2019-03-25 15:37:01 UTC
Description of problem:

Few words about environment:

- customer is running RHOSP 10 environment that was originally installed as RHOSP 8 and upgraded;
- customer has multiple cinder backends;
- """for i in `rpm -qa | grep cinder`; do echo $i; sudo rpm -V "$i"; done""" command didn't display any modifications of shipped cinder-volume code (customer replaced files in "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nexenta/" folder, but it shouldn't cause this problem).

I have analysed cinder-volume logs and found that the following messages were generated:

2018-09-25 15:34:18.784 381940 INFO cinder.api.openstack.wsgi [req-ac9fc8ce-d7c9-46c6-8e97-719c15f1754e 5
ebbe90c089647d9a5a9ad55cb67fed5 1058aeb310b944929945a7b177fd66f3 - default default] GET https://cloud.example.com:8776/v2/1058aeb310b944929945a7b177fd66f3/qos-specs/0a11db89-e787-49c6-98ff-7705105dd20
e/associations
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault [req-ac9fc8ce-d7c9-46c6-8e97-719c15f1754
e 5ebbe90c089647d9a5a9ad55cb67fed5 1058aeb310b944929945a7b177fd66f3 - default default] Caught error: <typ
e 'exceptions.ValueError'> A string is required in field projects[0], not a VolumeTypeProjects
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault Traceback (most recent call last):
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/middleware/fault.py", line 79, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return req.get_response(self.application)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 108, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return request.get_response(self.application)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     response = req.get_response(self._app)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     response = self.app(environ, start_response)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 843, in __call__
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     content_type, body, accept)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 902, in _process_stack
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     action_result = self.dispatch(meth, request, action_args)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 997, in dispatch
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return method(req=request, **action_args)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/contrib/qos_specs_manage.py", line 234, in associations
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     associates = qos_specs.get_associations(context, id)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/volume/qos_specs.py", line 139, in get_associations
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     qos_specs_id)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/objects/volume_type.py", line 146, in get_all_types_for_qos
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     types)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 1125, in obj_make_list
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     **extra_args)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/objects/volume_type.py", line 87, in _from_db_object
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     type.projects = db_type.get('projects', [])
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     field_value = field.coerce(self, name, value)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 195, in coerce
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return self._type.coerce(obj, attr, value)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 637, in coerce
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     coerced_list.extend(value)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 1190, in extend
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     for index, item in enumerate(t)]
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 1170, in _coerce_item
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return self._element_type.coerce(self._obj, att_name, item)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 195, in coerce
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     return self._type.coerce(obj, attr, value)
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 265, in coerce
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault     {'attr': attr, 'type': type(value).__name__})
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault ValueError: A string is required in field projects[0], not a VolumeTypeProjects
2018-09-25 15:34:18.797 381940 ERROR cinder.api.middleware.fault 
2018-09-25 15:34:18.802 381940 INFO cinder.api.middleware.fault [req-ac9fc8ce-d7c9-46c6-8e97-719c15f1754e 5ebbe90c089647d9a5a9ad55cb67fed5 1058aeb310b944929945a7b177fd66f3 - default default] https://cloud.example.com:8776/v2/1058aeb310b944929945a7b177fd66f3/qos-specs/0a11db89-e787-49c6-98ff-7705105dd20e/associations returned with HTTP 500
2018-09-25 15:34:18.804 381940 INFO eventlet.wsgi.server [req-ac9fc8ce-d7c9-46c6-8e97-719c15f1754e 5ebbe90c089647d9a5a9ad55cb67fed5 1058aeb310b944929945a7b177fd66f3 - default default] 192.168.1.1 "GET /v2/1058aeb310b944929945a7b177fd66f3/qos-specs/0a11db89-e787-49c6-98ff-7705105dd20e/associations HTTP/1.1" status: 500  len: 401 time: 0.0545561


Troubleshooting steps already taken:

- I have thought about DB issue, but DB dump doesn't contain any problem and is similar to DB in my lab.
- I have thought about code customizations, but it doesn't look like a cause of this issue
- I have thought about bug in old cinder package, but the problem is still there after minor upgrade.

At this point I need help from engineering to identify root cause and solve this issue. I will provide more information privately.


Note You need to log in before you can comment on or make changes to this bug.