Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1055660 - qpidd only binds to ipv6 socket after reboot
Summary: qpidd only binds to ipv6 socket after reboot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: qpid-cpp
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Darryl L. Pierce
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1049488 (view as bug list)
Depends On:
Blocks: 1049504
TreeView+ depends on / blocked
 
Reported: 2014-01-20 17:41 UTC by Lars Kellogg-Stedman
Modified: 2015-06-22 00:08 UTC (History)
7 users (show)

Fixed In Version: qpid-cpp-0.24-6.fc19
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-01 03:59:50 UTC


Attachments (Terms of Use)

Description Lars Kellogg-Stedman 2014-01-20 17:41:14 UTC
Description of problem:

After rebooting, qpidd is only listening on an IPv6 socket:

# ps -fe | grep qpid
qpidd      276     1  0 17:11 ?        00:00:00 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

# netstat -tlnp | grep 276
tcp6       0      0 :::5672                 :::*                    LISTEN      276/qpidd           

This makes the service unavailable to clients connecting via IPv4, leading to errors such as this from nova-compute:

2014-01-20 17:37:42.487 1703 ERROR nova.openstack.common.rpc.impl_qpid [req-8a3cbe13-ce56-47bb-982b-a8bf1e23a619 None None] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds

Restarting the qpidd service corrects the issue:

# systemctl restart qpidd
# ps -fe | grep qpid
qpidd     3093     1  2 17:38 ?        00:00:00 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf
root      3112  2847  0 17:38 pts/0    00:00:00 grep --color=auto qpid
# netstat -tlnp | grep 3093
tcp        0      0 0.0.0.0:5672            0.0.0.0:*               LISTEN      3093/qpidd          
tcp6       0      0 :::5672                 :::*                    LISTEN      3093/qpidd          


Version-Release number of selected component (if applicable):

qpid-cpp-server-0.24-8.fc20.x86_64

How reproducible:

This seems to happen reliably on reboot.

Comment 1 Darryl L. Pierce 2014-01-20 18:23:32 UTC
Can you share the contents of your /etc/qpid/qpidd.conf file? Also any changes the configuration for the service that have been made.

Comment 2 Lars Kellogg-Stedman 2014-01-20 21:40:14 UTC
The contents of /etc/qpid/qpidd.conf are:

    log-enable=info+

I have made no changes to the service.

Whether or not this problems happens seems to be a bit of a crapshoot.  Sometimes I reboot to find qpid listening on both IPv4 and IPv6 sockets, sometimes just IPv6.

With info+ logging enabled, when qpid listens on both interfaces it logs:

Jan 20 21:38:03 lars-devstack.os1 qpidd[666]: 2014-01-20 21:38:03 [Network] info Listening to: 0.0.0.0:5672
Jan 20 21:38:03 lars-devstack.os1 qpidd[666]: 2014-01-20 21:38:03 [Network] info Listening to: [::]:5672
Jan 20 21:38:03 lars-devstack.os1 qpidd[666]: 2014-01-20 21:38:03 [Network] notice Listening on TCP/TCP6 port 5672


While when it only listens on the IPv6 socket it only logs:

Jan 20 21:32:39 lars-devstack.os1 qpidd[359]: 2014-01-20 21:32:39 [Network] info Listening to: [::]:5672
Jan 20 21:32:39 lars-devstack.os1 qpidd[359]: 2014-01-20 21:32:39 [Network] notice Listening on TCP/TCP6 port 5672

Comment 3 Andrew Stitcher 2014-01-20 23:21:28 UTC
I suspect this is a function of parallel startup:

By default qpidd will actively look for all the active interfaces when it starts. It is very possible that when qpidd is started at boot the network inerfaces aren't all up yet.

It should be possible to make qpidd startup dependent on the appropriate interfaces receiving an IP address and only then being started.

So in order to work on simple systems you might need to wait for IPv6/IPv4 addresses to be available on the loopback interface and IPv4/IPv6 addresses to be available on the em1 interface.

Comment 4 Andrew Stitcher 2014-01-20 23:23:46 UTC
To clarify a little, qpidd doesn't bind to the catch all address and so will not notice if new network interfaces arrive whilst it is running.

Comment 5 Lars Kellogg-Stedman 2014-01-21 02:18:00 UTC
It *appears* to be binding to the catch all address.  Or do you mean some other than `0.0.0.0`?  I added an `ExecStartPre` to the `qpidd.service` file to capture the network state when the service starts, and there did not appear to be any correlation between the state of `eth0` and successfully binding to the IPv4 socket.

That is, if the `ExecStartPre` script ran `ip addr`, on a *successful* startup (qpidd bound to both the IPv4 and IPv6 sockets), the output of `ip addr` was:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

That is, `eth0` didn't even show up in the output.  At other times, the output look like this:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether fa:16:3e:41:e7:92 brd ff:ff:ff:ff:ff:ff

But I haven't been able to spot a correlation between either state and the problem.  It certainly looks like a race condition *somewhere*.

Comment 6 Darryl L. Pierce 2014-01-21 15:09:19 UTC
(In reply to Andrew Stitcher from comment #3)
> I suspect this is a function of parallel startup:
> 
> By default qpidd will actively look for all the active interfaces when it
> starts. It is very possible that when qpidd is started at boot the network
> inerfaces aren't all up yet.
> 
> It should be possible to make qpidd startup dependent on the appropriate
> interfaces receiving an IP address and only then being started.
> 
> So in order to work on simple systems you might need to wait for IPv6/IPv4
> addresses to be available on the loopback interface and IPv4/IPv6 addresses
> to be available on the em1 interface.

Hrm, that shouldn't be the case. The systemd service file for qpidd has:

  After=network.target

so it shouldn't be starting in parallel to but after the network service has started.

Comment 7 Lars Kellogg-Stedman 2014-01-21 15:23:49 UTC
Actually, the service file for qpid-cpp-server-0.24-8.fc20.x86_64 has:

[Unit]
Description=An AMQP message broker daemon.
Documentation=man:qpidd(1) http://qpid.apache.org/
Requires=network.target

That is, it has a Requires dependency, but not an After dependency.  Let me add that and see what happens.

Comment 8 Darryl L. Pierce 2014-01-21 15:25:00 UTC
(In reply to Lars Kellogg-Stedman from comment #7)
> Actually, the service file for qpid-cpp-server-0.24-8.fc20.x86_64 has:
> 
> [Unit]
> Description=An AMQP message broker daemon.
> Documentation=man:qpidd(1) http://qpid.apache.org/
> Requires=network.target
> 
> That is, it has a Requires dependency, but not an After dependency.  Let me
> add that and see what happens.

CORRECTION: This change is in the 0.26 update that will go out. I'm going to go ahead an push an updated 0.24 with the fixed services file since 0.26 isn't up to a vote yet for release.

Comment 9 Lars Kellogg-Stedman 2014-01-21 15:32:59 UTC
That should do it.  Empirical evidence suggests that adding the After constraint fixes the problem.

Comment 10 Andrew Stitcher 2014-01-22 17:26:15 UTC
(In reply to Lars Kellogg-Stedman from comment #5)
> It *appears* to be binding to the catch all address.  Or do you mean some
> other than `0.0.0.0`? 

I misspoke - If you specify an interface to bind to it will enumerate the interfaces and only bind to addresses in can see on interfaces. Here where you are not specifying any interface it *will* bind to the catchall addresses that it finds. However if networking is not fully up yet then it won't necessarily find all the catch all addresses.

Comment 11 Fedora Update System 2014-01-22 19:53:34 UTC
qpid-cpp-0.24-9.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/qpid-cpp-0.24-9.fc20

Comment 12 Fedora Update System 2014-01-23 10:40:02 UTC
qpid-cpp-0.24-6.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/qpid-cpp-0.24-6.fc19

Comment 13 Fedora Update System 2014-01-23 11:04:18 UTC
Package qpid-cpp-0.24-9.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing qpid-cpp-0.24-9.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-1366/qpid-cpp-0.24-9.fc20
then log in and leave karma (feedback).

Comment 14 Fedora Update System 2014-02-01 03:59:50 UTC
qpid-cpp-0.24-9.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 15 Fedora Update System 2014-02-01 04:06:03 UTC
qpid-cpp-0.24-6.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 16 Steven Hardy 2014-05-20 09:40:40 UTC
*** Bug 1049488 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.