Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1366348 - Recent Update to initscripts Causing OpenStack Bridge To Fail
Summary: Recent Update to initscripts Causing OpenStack Bridge To Fail
Keywords:
Status: CLOSED DUPLICATE of bug 1367580
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: NetworkManager
Version: 7.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Rashid Khan
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-11 17:49 UTC by Dan Sneddon
Modified: 2016-08-17 18:23 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-17 18:23:17 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1364583 None CLOSED rhel-osp-director: upgrade from 7.3 ->8.0 , br-ex on the controllers was missing ip address and pacemaker was unable to ... 2018-09-26 04:39:57 UTC

Internal Links: 1367580

Description Dan Sneddon 2016-08-11 17:49:32 UTC
Description of problem:
After a recent update to initscripts, upgrades to Red Hat OpenStack Platform began failing. This was traced to a reproducible bug where the bridge named "br-ex" would be deactivated shortly after activation. This appears to be caused by initscripts+NetworkManager, because when we deactivate NetworkManager everything works. 

Version-Release number of selected component (if applicable):
I think the version number of the new initscripts (released in the last two weeks) is: 9.49.30-1.el7_2.3

How reproducible:
100%

Steps to Reproduce:
1. Deploy Red Hat OpenStack Platform 7 (succeeds)
2. Upgrade to Red Hat OpenStack Platform 8 (fails)

Actual results:
The deployment fails because the br-ex bridge is brought down shortly after activation. We see in the logs that NetworkManager reads the configuration file and sees that NM_CONTROLLED=no, but immediately after the bridge is marked as unmanaged it is shut down. The log message appears to indicate that NetworkManager is involved:
Aug  9 20:44:44 overcloud-controller-1 nm-dispatcher: Dispatching action 'down' for br-ex

Expected results:
Prior to this latest initscripts update, the upgrade would succeed and the br-ex bridge would remain up.

Additional info:
We believe that this is the offending patch that is causing the failure:

http://pkgs.devel.redhat.com/cgit/rpms/initscripts/commit/?h=rhel-7.2&id=4aeb2f7ee2b31630ae5ff27e8046b5117b7f7a22

Here are the /var/log/messages logs from immediately after the bridge interface was brought up via "ifup br-ex". There is more info in the linked BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=1364583

Aug  9 20:44:44 overcloud-controller-1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex em1 -- add-port br-ex em1
Aug  9 20:44:44 overcloud-controller-1 NetworkManager[775]: <info>  (em1): enslaved to non-master-type device ovs-system; ignoring
Aug  9 20:44:44 overcloud-controller-1 kernel: device em1 entered promiscuous mode
Aug  9 20:44:44 overcloud-controller-1 NetworkManager[775]: <info>  ifcfg-rh: new connection /etc/sysconfig/network-scripts/ifcfg-br-ex (f0123855-5f72-fb68-339e-ef4f1d038014,"System br-ex")
Aug  9 20:44:44 overcloud-controller-1 NetworkManager[775]: <warn>  ifcfg-rh: Ignoring connection /etc/sysconfig/network-scripts/ifcfg-br-ex (f0123855-5f72-fb68-339e-ef4f1d038014,"System br-
ex") / device 'br-ex' due to NM_CONTROLLED=no.
Aug  9 20:44:44 overcloud-controller-1 NetworkManager[775]: <info>  (br-ex): device state change: activated -> unmanaged (reason 'unmanaged') [100 10 3]
Aug  9 20:44:44 overcloud-controller-1 NetworkManager[775]: <info>  NetworkManager state is now CONNECTED_LOCAL
Aug  9 20:44:44 overcloud-controller-1 NetworkManager[775]: <info>  NetworkManager state is now DISCONNECTED
Aug  9 20:44:44 overcloud-controller-1 kernel: IPv6: ADDRCONF(NETDEV_UP): br-ex: link is not ready
Aug  9 20:44:44 overcloud-controller-1 NetworkManager[775]: <info>  (br-ex): link disconnected
Aug  9 20:44:44 overcloud-controller-1 dbus-daemon: dbus[758]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
Aug  9 20:44:44 overcloud-controller-1 dbus[758]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
Aug  9 20:44:44 overcloud-controller-1 systemd: Starting Network Manager Script Dispatcher Service...
Aug  9 20:44:44 overcloud-controller-1 dbus[758]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Aug  9 20:44:44 overcloud-controller-1 systemd: Started Network Manager Script Dispatcher Service.
Aug  9 20:44:44 overcloud-controller-1 dbus-daemon: dbus[758]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Aug  9 20:44:44 overcloud-controller-1 nm-dispatcher: Dispatching action 'down' for br-ex

Comment 2 Lukáš Nykrýn 2016-08-12 07:34:56 UTC
That patch should be pretty harmless, on the initscripts side it only adds a call of  nmcli con load "/etc/sysconfig/network-scripts/$CONFIG" for every device. We could workaround it by skipping that for bridges, but maybe there is a better way to fix it on the NM side.

Comment 3 Thomas Haller 2016-08-17 18:23:17 UTC

*** This bug has been marked as a duplicate of bug 1367580 ***


Note You need to log in before you can comment on or make changes to this bug.