Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1060711 - neutron qpid reconnection delay must be more accurate
Summary: neutron qpid reconnection delay must be more accurate
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 4.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z4
: 4.0
Assignee: Ihar Hrachyshka
QA Contact: yfried
URL:
Whiteboard:
Depends On:
Blocks: RHEL-OSP_Neutron_HA
TreeView+ depends on / blocked
 
Reported: 2014-02-03 12:46 UTC by Fabio Massimo Di Nitto
Modified: 2016-04-26 14:16 UTC (History)
9 users (show)

Fixed In Version: openstack-neutron-2013.2.3-3.el6ost
Doc Type: Bug Fix
Doc Text:
Cause: The Qpid reconnection delay was fixed to 60 seconds. Consequence: In HA environments, if the Qpid service is not available at connection time, the client will take 60 seconds before trying to reconnect again. Fix: Changed the default reconnection retry time to 5 seconds for 4.0. Result: If Qpid is unavailable at connection time, it will retry in 5 seconds.
Clone Of: 1060689
Environment:
Last Closed: 2014-05-29 20:19:02 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1281148 None None None Never
Red Hat Product Errata RHSA-2014:0516 normal SHIPPED_LIVE Moderate: openstack-neutron security, bug fix, and enhancement update 2014-05-30 00:15:59 UTC
OpenStack gerrit 80998 None None None Never
OpenStack gerrit 81009 None None None Never
OpenStack gerrit 82786 None None None Never

Description Fabio Massimo Di Nitto 2014-02-03 12:46:24 UTC
+++ This bug was initially created as a clone of Bug #1060689 +++

The current loop is:

        delay = 1
        while True:
            # Close the session if necessary
            if self.connection.opened():
                try:
                    self.connection.close()
                except qpid_exceptions.ConnectionError:
                    pass

            broker = self.brokers[attempt % len(self.brokers)]
            attempt += 1

            try:
                self.connection_create(broker)
                self.connection.open()
            except qpid_exceptions.ConnectionError, e:
                msg_dict = dict(e=e, delay=delay)
                msg = _("Unable to connect to AMQP server: %(e)s. "
                        "Sleeping %(delay)s seconds") % msg_dict
                LOG.error(msg)
                time.sleep(delay)
                delay = min(2 * delay, 60)

that can lead to over 60 seconds waiting time if the qpid sever is not immediately available at startup.

60 seconds is too long for HA environment where timers need to be very aggressive to reduce downtime to the very minimum.

This is a blocker for HA deployments.

Comment 1 Ihar Hrachyshka 2014-02-03 13:07:31 UTC
Note: this patch, if implemented, won't go to upstream since oslo-rpc that we use is for bug fixing only, and this patch will be considered as too 'featurey'. So we would need to support our own downstream patch for each service that use oslo-rpc if we want to see this in current release.

Comment 4 yfried 2014-04-22 09:12:23 UTC
RHOS 4.0 on RHEL6.5

python-neutron-2013.2.3-4.el6ost.noarch
python-neutronclient-2.3.4-1.el6ost.noarch
openstack-neutron-openvswitch-2013.2.3-4.el6ost.noarch
openstack-neutron-2013.2.3-4.el6ost.noarch

killed qpid service and checked log:
2014-04-22 12:07:21.221 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 1 seconds
2014-04-22 12:07:22.222 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 2 seconds
2014-04-22 12:07:22.438 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 3 seconds
2014-04-22 12:07:22.439 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 3 seconds
2014-04-22 12:07:24.223 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 3 seconds
2014-04-22 12:07:25.439 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 4 seconds
2014-04-22 12:07:25.440 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 4 seconds
2014-04-22 12:07:27.224 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 4 seconds
2014-04-22 12:07:29.440 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 5 seconds
2014-04-22 12:07:29.441 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 5 seconds
2014-04-22 12:07:31.226 20972 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 5 seconds

Comment 6 errata-xmlrpc 2014-05-29 20:19:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0516.html


Note You need to log in before you can comment on or make changes to this bug.