Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1511041 - Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
Summary: Failed to consume message from queue: 'NoneType' object has no attribute 'ite...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-amqp
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 12.0 (Pike)
Assignee: John Eckersberg
QA Contact: Udi Shkalim
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-08 15:23 UTC by Sai Sindhur Malleni
Modified: 2018-02-05 19:15 UTC (History)
10 users (show)

Fixed In Version: python-amqp-2.1.4-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-13 22:20:31 UTC


Attachments (Terms of Use)
Copy of https://gist.github.com/anonymous/eda3c3589806eddcbff108e76feea29a (deleted)
2017-11-08 15:45 UTC, Victor Stinner
no flags Details


Links
System ID Priority Status Summary Last Updated
Launchpad 1504725 None None None 2017-11-08 18:47:31 UTC
Red Hat Product Errata RHEA-2017:3462 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Description Sai Sindhur Malleni 2017-11-08 15:23:43 UTC
Description of problem:
Seeing 100s of errors like this in the neutron logs
https://gist.github.com/anonymous/eda3c3589806eddcbff108e76feea29a
Version-Release number of selected component (if applicable):
OSP 12
Puddle: 2017-10-31.2

How reproducible:
100%

Steps to Reproduce:
1. Create 100s of neutron resources
2.
3.

Actual results:
Seeing this error in the neutron-server logs

Expected results:
No Errors

Additional info:

Comment 2 John Eckersberg 2017-11-08 18:47:32 UTC
This looks similar/same as https://bugs.launchpad.net/oslo.messaging/+bug/1504725, adding tracker.

Comment 3 John Eckersberg 2017-11-08 22:12:54 UTC
Possibly this issue?

https://github.com/celery/py-amqp/pull/121

Comment 4 Victor Stinner 2017-11-13 10:47:29 UTC
It looks like this bug which was fixed in python-amqp 2.2.0: https://github.com/celery/py-amqp/pull/128

Comment 5 Marian Krcmarik 2017-11-13 11:05:39 UTC
I could reproduce the bug several times on OSP12 containerized deployment, specifically I see the traceback on nova compute log and then I am not able to run an instance on such compute node unless I restart nova compute service/container so in my case it has fatal consequences.
I am triggering the problem by running recovery tests which ungracefully reset controller nodes, especially rabbitmq servers which is relevant to the bug I assume.

Comment 6 Eran Kuris 2017-11-13 12:39:19 UTC
I could reproduce the same issue on OSP-12 with OVN.
During HA recovery test, after the system was fully recovered I failed to boot new VM.
I left the system for few minutes and after that, I could launch new VM.

Comment 7 Eran Kuris 2017-11-13 12:40:32 UTC
(In reply to Eran Kuris from comment #6)
> I could reproduce the same issue on OSP-12 with OVN.
> During HA recovery test, after the system was fully recovered I failed to
> boot new VM.
> I left the system for few minutes and after that, I could launch new VM.

PS 
I saw this error also on nova-scheduler.log & Neutron server-log.

Comment 8 John Eckersberg 2017-11-13 14:59:46 UTC
(In reply to Victor Stinner from comment #4)
> It looks like this bug which was fixed in python-amqp 2.2.0:
> https://github.com/celery/py-amqp/pull/128

+1, this is exactly the problem.

Comment 11 Udi Shkalim 2017-11-20 15:26:08 UTC
Verified on: python-amqp-2.1.4-2.el7ost.noarch

Ran HA recovery tests and was able to boot an instance with no reoccurring messages in the logs mentioned

Comment 15 errata-xmlrpc 2017-12-13 22:20:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462


Note You need to log in before you can comment on or make changes to this bug.