Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1595206 - [UPDATE] After update cinder-scheduler keep exiting with error "Failed to declare consumer for topic 'cinder-schedul er'"
Summary: [UPDATE] After update cinder-scheduler keep exiting with error "Failed to dec...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: John Eckersberg
QA Contact: Avi Avraham
Kim Nylander
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-26 11:14 UTC by Raviv Bar-Tal
Modified: 2018-07-24 08:54 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-24 08:54:33 UTC
Target Upstream Version:


Attachments (Terms of Use)
cinder-scheduler.log (deleted)
2018-06-26 11:16 UTC, Raviv Bar-Tal
no flags Details
controller-0 /var/log/container (deleted)
2018-06-26 12:15 UTC, Raviv Bar-Tal
no flags Details

Description Raviv Bar-Tal 2018-06-26 11:14:07 UTC
Description of problem:
openstack-cinder-scheduler docker keeps restarting and exiting after ~2 minutes with this error:
2018-06-26 11:06:20.674 1 ERROR oslo.messaging._drivers.impl_rabbit [req-3b52dc4a-84f7-4f6b-bca6-fd5b47583a72 - - - - -] Failed to declare consumer for topic 'cinder-schedul
er': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed 
to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout
2018-06-26 11:06:20.675 1 ERROR oslo.messaging._drivers.impl_rabbit [req-3b52dc4a-84f7-4f6b-bca6-fd5b47583a72 - - - - -] Unable to connect to AMQP server on controller-0.int
ernalapi.localdomain:5672 after None tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout: NotFound: Q
ueue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout
2018-06-26 11:06:20.676 1 ERROR oslo_service.service [req-3b52dc4a-84f7-4f6b-bca6-fd5b47583a72 - - - - -] Error starting thread.: MessageDeliveryFailure: Unable to connect t
o AMQP server on controller-0.internalapi.localdomain:5672 after None tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhos
t '/' due to timeout
2018-06-26 11:06:20.676 1 ERROR oslo_service.service Traceback (most recent call last):
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 729, in run_service
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     service.start()
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 239, in start
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     self.rpcserver.start()
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 270, in wrapper
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     log_after, timeout_timer)
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 190, in run_once
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     post_fn = fn()
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 269, in <lambda>
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     states[state].run_once(lambda: fn(self, *args, **kwargs),
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 416, in start
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     self.listener = self._create_listener()
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 151, in _create_listener
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     return self.transport._listen(self._target, 1, None)
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 146, in _listen
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     batch_timeout)
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 572, in listen
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     callback=listener)
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1120, in declare_topic_consumer
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     self.declare_consumer(consumer)
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1023, in declare_consumer
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     error_callback=_connect_error)
2018-06-26 11:06:20.676 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 825, in ensure
2018-06-26 11:06:20.676 1 ERROR oslo_service.service     raise exceptions.MessageDeliveryFailure(msg)
2018-06-26 11:06:20.676 1 ERROR oslo_service.service MessageDeliveryFailure: Unable to connect to AMQP server on controller-0.internalapi.localdomain:5672 after None tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout
2018-06-26 11:06:20.676 1 ERROR oslo_service.service 
2018-06-26 11:06:20.679 1 CRITICAL cinder [req-831337c4-ce4b-4877-97d2-01da4fb7dc9d - - - - -] Unhandled error: AttributeError: 'NoneType' object has no attribute 'cleanup'
2018-06-26 11:06:20.679 1 ERROR cinder Traceback (most recent call last):
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/bin/cinder-scheduler", line 10, in <module>
2018-06-26 11:06:20.679 1 ERROR cinder     sys.exit(main())
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/cinder/cmd/scheduler.py", line 55, in main
2018-06-26 11:06:20.679 1 ERROR cinder     service.wait()
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 629, in wait
2018-06-26 11:06:20.679 1 ERROR cinder     _launcher.wait()
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 334, in wait
2018-06-26 11:06:20.679 1 ERROR cinder     super(ServiceLauncher, self).wait()
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 237, in wait
2018-06-26 11:06:20.679 1 ERROR cinder     self.services.wait()
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in wait
2018-06-26 11:06:20.679 1 ERROR cinder     service.wait()
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 442, in wait
2018-06-26 11:06:20.679 1 ERROR cinder     self.rpcserver.wait()
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 270, in wrapper
2018-06-26 11:06:20.679 1 ERROR cinder     log_after, timeout_timer)
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 190, in run_once
2018-06-26 11:06:20.679 1 ERROR cinder     post_fn = fn()
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 269, in <lambda>
2018-06-26 11:06:20.679 1 ERROR cinder     states[state].run_once(lambda: fn(self, *args, **kwargs),
2018-06-26 11:06:20.679 1 ERROR cinder   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 448, in wait
2018-06-26 11:06:20.679 1 ERROR cinder     self.listener.cleanup()
2018-06-26 11:06:20.679 1 ERROR cinder AttributeError: 'NoneType' object has no attribute 'cleanup'
2018-06-26 11:06:20.679 1 ERROR cinder 



Version-Release number of selected component (if applicable):

puppet-cinder-12.4.1-0.20180329071637.4011a82.el7ost.noarch
python2-cinderclient-3.5.0-1.el7ost.noarch
python-cinder-12.0.1-0.20180418194614.c476898.el7ost.noarch
openstack-cinder-12.0.1-0.20180418194614.c476898.el7ost.noarch

How reproducible:


Steps to Reproduce:
1.Install osp13 beta puddle
2. update to 2018-06-21.2 puddle
3.check cinder-scheduler on the controller ('docker ps | grep cinder-scheduler')

Actual results:
The docker is restarting, remains in "Up" (running) status for 2 minutes and then restating again.
root@controller-0 ~]# docker ps | grep cinder
89ddacb25105        192.168.24.1:8787/rhosp13/openstack-cinder-api:2018-06-21.2                  "kolla_start"            17 hours ago        Up 17 hours                                           cinder_api_cron
e2f9a7d9395b        192.168.24.1:8787/rhosp13/openstack-cinder-scheduler:2018-06-21.2            "kolla_start"            17 hours ago        Up 37 seconds (healthy)                               cinder_scheduler
c8e4abfe8344        192.168.24.1:8787/rhosp13/openstack-cinder-api:2018-06-21.2                  "kolla_start"            17 hours ago        Up 17 hours                                           cinder_api


Expected results:


Additional info:

Comment 1 Raviv Bar-Tal 2018-06-26 11:16:45 UTC
Created attachment 1454623 [details]
cinder-scheduler.log

Comment 2 Alan Bishop 2018-06-26 12:10:24 UTC
The log shows issues with cinder connecting to rabbitmq, so moving this to PIDONE. As a side note, I believe we don't support updating beta releases to RC/GA, but I'm not sure if that's relevant here.

Comment 3 Raviv Bar-Tal 2018-06-26 12:15:20 UTC
Created attachment 1454630 [details]
controller-0 /var/log/container

Comment 6 John Eckersberg 2018-07-10 14:30:47 UTC
Can you provide the full rabbitmq logs?  The sosreports from the controllers only have the tailed logs.

Comment 7 Raviv Bar-Tal 2018-07-24 08:54:33 UTC
I did not manage to reproduce this bug with newer puddle (2018-07-13.1),
I assume this was fix unintentionally


Note You need to log in before you can comment on or make changes to this bug.