Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1353971 - 'Failed to expunge metric: PermissionError: Failed to remove metric' with ceph backend
Summary: 'Failed to expunge metric: PermissionError: Failed to remove metric' with ce...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ga
: 9.0 (Mitaka)
Assignee: Giulio Fidente
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On:
Blocks: 1356107
TreeView+ depends on / blocked
 
Reported: 2016-07-08 14:49 UTC by Yurii Prokulevych
Modified: 2016-08-18 04:00 UTC (History)
14 users (show)

Fixed In Version: openstack-puppet-modules-8.1.7-2.el7ost
Doc Type: Bug Fix
Doc Text:
The Ceph puppet module (puppet-ceph) did not update CephX keyrings when the caps parameter for a key were changed. This caused overcloud upgrades to fail, as caps to operate on the new 'metrics' pool were not added to the CephX keyring. With this update, puppet-ceph regenerates the virsh secret or updates its key if either 'rbd_keyring' or 'linvirt_rbd_secret_uuid' change. This ensures that the CephX keyring is updated as required when secrets or caps change.
Clone Of:
Environment:
Last Closed: 2016-08-15 07:19:01 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1606918 None None None 2016-07-27 13:06:31 UTC
OpenStack gerrit 232369 None None None 2016-07-29 14:12:26 UTC
OpenStack gerrit 350512 None None None 2016-08-03 10:48:45 UTC
Red Hat Product Errata RHEA-2016:1597 normal SHIPPED_LIVE Red Hat OpenStack Platform 9 Release Candidate Advisory 2016-08-11 16:06:52 UTC

Description Yurii Prokulevych 2016-07-08 14:49:51 UTC
Description of problem:
Gnocchi manually configured to use ceph backend.
Package python-cradox-1.1.8-2.el7ost.x86_64 installed manually.

Attemtp to add measure to metric fails:
gnocchi  measures add -m '2016-07-07T14:45:01@1' 8d4c4821-e69d-497a-a9c5-db88f78aaba9
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
 (HTTP 500)

Excerpt from gnocchi/metricd.log

2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage [-] Unable to expunge metric f9633d99-eb1c-400a-b271-b4d82dd83386 from storage
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage Traceback (most recent call last):
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 204, in expunge_metrics
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage     self.delete_metric(m, sync)
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 250, in delete_metric
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage     self._delete_metric_archives(metric)
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 360, in _delete_metric_archives
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage     str("gnocchi_%s_%s" % (metric.id, aggregation)))
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "cradox.pyx", line 413, in cradox.requires.wrapper.validate_func (cradox.c:4188)
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "cradox.pyx", line 2352, in cradox.Ioctx.remove_object (cradox.c:29337)
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage PermissionError: Failed to remove 'gnocchi_f9633d99-eb1c-400a-b271-b4d82dd83386_max'

Excerpt from httpd/gnocchi_wsgi_error.log
[Fri Jul 08 13:44:50.061870 2016] [:error] [pid 16248] [remote 192.0.2.20:4] mod_wsgi (pid=16248): Exception occurred processing WSGI script '/var/www/cgi-bin/gnocchi/app'.
[Fri Jul 08 13:44:50.061919 2016] [:error] [pid 16248] [remote 192.0.2.20:4] Traceback (most recent call last):
[Fri Jul 08 13:44:50.061936 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__
[Fri Jul 08 13:44:50.062065 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return app(environ, start_response)
[Fri Jul 08 13:44:50.062075 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
[Fri Jul 08 13:44:50.062089 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     resp = self.call_func(req, *args, **self.kwargs)
[Fri Jul 08 13:44:50.062096 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
[Fri Jul 08 13:44:50.062106 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return self.func(req, *args, **kwargs)
[Fri Jul 08 13:44:50.062113 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
[Fri Jul 08 13:44:50.062159 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     response = req.get_response(self._app)
[Fri Jul 08 13:44:50.062168 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send
[Fri Jul 08 13:44:50.062180 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     application, catch_exc_info=False)
[Fri Jul 08 13:44:50.062187 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
[Fri Jul 08 13:44:50.062210 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     app_iter = application(self.environ, start_response)
[Fri Jul 08 13:44:50.062218 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/exc.py", line 1117, in __call__
[Fri Jul 08 13:44:50.062228 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return self.application(environ, start_response)
[Fri Jul 08 13:44:50.062235 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/rest/app.py", line 68, in __call__
[Fri Jul 08 13:44:50.062247 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return self.app(environ, start_response)
[Fri Jul 08 13:44:50.062254 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
[Fri Jul 08 13:44:50.062266 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return self.application(environ, start_response)
[Fri Jul 08 13:44:50.062272 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 829, in __call__
[Fri Jul 08 13:44:50.062282 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return super(Pecan, self).__call__(environ, start_response)
[Fri Jul 08 13:44:50.062289 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__
[Fri Jul 08 13:44:50.062308 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     self.invoke_controller(controller, args, kwargs, state)
[Fri Jul 08 13:44:50.062317 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 572, in invoke_controller
[Fri Jul 08 13:44:50.062328 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     result = controller(*args, **kwargs)
[Fri Jul 08 13:44:50.062335 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/rest/__init__.py", line 550, in get_measures
[Fri Jul 08 13:44:50.062346 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     float(granularity) if granularity is not None else None)
[Fri Jul 08 13:44:50.062353 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 125, in get_measures
[Fri Jul 08 13:44:50.062363 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     from_timestamp, to_timestamp)]
[Fri Jul 08 13:44:50.062370 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 156, in _get_measures_timeserie
[Fri Jul 08 13:44:50.062380 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     metric, aggregation, granularity)
[Fri Jul 08 13:44:50.062386 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 304, in _list_split_keys_for_metric
[Fri Jul 08 13:44:50.062397 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     xattrs = ioctx.get_xattrs("gnocchi_%s_container" % metric.id)
[Fri Jul 08 13:44:50.062403 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "cradox.pyx", line 413, in cradox.requires.wrapper.validate_func (cradox.c:4188)
[Fri Jul 08 13:44:50.062515 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "cradox.pyx", line 2463, in cradox.Ioctx.get_xattrs (cradox.c:30504)
[Fri Jul 08 13:44:50.062595 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "cradox.pyx", line 1341, in cradox.XattrIterator.__cinit__ (cradox.c:16886)
[Fri Jul 08 13:44:50.062675 2016] [:error] [pid 16248] [remote 192.0.2.20:4] PermissionError: Failed to get rados xattrs for object 'gnocchi_4b77de95-893d-4e62-b99f-b2121697aa36_container'


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-gnocchi-statsd-2.1.3-2.el7ost.noarch
python-gnocchi-2.1.3-2.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-2.1.3-2.el7ost.noarch
openstack-gnocchi-carbonara-2.1.3-2.el7ost.noarch
openstack-gnocchi-metricd-2.1.3-2.el7ost.noarch
python-gnocchiclient-2.2.0-1.el7ost.noarch
openstack-gnocchi-common-2.1.3-2.el7ost.noarch
openstack-gnocchi-api-2.1.3-2.el7ost.noarch

python-cradox-1.1.8-2.el7ost.x86_64

Expected results:
-----------------
Gnocchi is handling measures for metrics without errors

Additional info:
Virtual environment: 3controllers + 2computes + 1ceph, after upgrade.

Comment 5 Pradeep Kilambi 2016-07-27 13:05:59 UTC
Turns out the issue is in puppet-ceph where if the secret is already there we dont update the acls and hence the key. See https://github.com/openstack/puppet-ceph/blob/master/manifests/key.pp#L145

gfidente will push a fix upstream.

Comment 6 Giulio Fidente 2016-08-02 22:59:28 UTC
The puppet-ceph submission is now merged, the puppet-nova submission is still failing on unrelated CI issues.

Comment 13 Yurii Prokulevych 2016-08-05 15:08:39 UTC
Gnocchi is working with ceph backend after the upgrade.

Package: openstack-puppet-modules-8.1.7-2.el7ost.noarch

Comment 15 errata-xmlrpc 2016-08-15 07:19:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html


Note You need to log in before you can comment on or make changes to this bug.