Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1100264 - [RHEVM] adding host with VLANed network using dhcp fails
Summary: [RHEVM] adding host with VLANed network using dhcp fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.4.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.5.0
Assignee: Barak
QA Contact: Martin Pavlik
URL:
Whiteboard: network
Depends On:
Blocks: rhev3.5beta 1156165
TreeView+ depends on / blocked
 
Reported: 2014-05-22 11:46 UTC by Martin Pavlik
Modified: 2016-02-10 19:48 UTC (History)
8 users (show)

Fixed In Version: vt1.3, 4.16.0-1.el6_5
Doc Type: Bug Fix
Doc Text:
When a virtual device such as a VLAN was created with a pre-existing associated ifcfg file, Udev would execute ifup on it. If the ifcfg file was configuring the VLAN device for DHCP network address management, when VDSM executed ifup for the device, dhclient would fail as another dhclient (the one started by the ifup Udev performed) would already be controlling the device. Now, hotplug=no has been added to the ifcfg file for virtual devices, so configuring VLANs with DHCP works properly.
Clone Of:
Environment:
Last Closed: 2015-02-11 21:11:08 UTC
oVirt Team: Network
Target Upstream Version:


Attachments (Terms of Use)
logs_from_host_with_failed_install (deleted)
2014-05-22 11:46 UTC, Martin Pavlik
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0159 normal SHIPPED_LIVE vdsm 3.5.0 - bug fix and enhancement update 2015-02-12 01:35:58 UTC
oVirt gerrit 27980 None None None Never
oVirt gerrit 28038 None None None Never
oVirt gerrit 28039 None None None Never

Description Martin Pavlik 2014-05-22 11:46:17 UTC
Created attachment 898352 [details]
logs_from_host_with_failed_install

Description of problem:
if engine uses VLAN tagged management network (rhevm) and host is being added over VLAN tagged interface using DHCP to obtain its IP, host install fails


Reproducer:
1) data center with bridgeless, VLAN tagged (I used VLANID=172) rhevm

2)
host:
install rhel6.5
set up interface em1.172 with DHCP

cat /etc/sysconfig/network-scripts/ifcfg-em1
DEVICE="em1"
BOOTPROTO="none"
IPV6INIT="no"
MTU="1500"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"

cat /etc/sysconfig/network-scripts/ifcfg-em1.172 
DEVICE=em1.172
ONBOOT=yes
VLAN=yes
BOOTPROTO=dhcp
DEFROUTE=yes
NM_CONTROLLED=no



3)
add host to rhevm

result: engine loses connectivity to host since the host loses IP


Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Manager Version: 3.4.0-0.21.el6ev 

How reproducible:
100%

Actual results:
install fails

Expected results:
install successful

Additional info:
there seem to be 2 problems

1) first is udev spawning dhclient for newly defined em1.172, this can be solved by adding HOTPLUG=no in ifcfg file apuimendo already posted a patch for this http://gerrit.ovirt.org/27980


cat /etc/sysconfig/network-scripts/ifcfg-em1.172 
DEVICE=em1.172
ONBOOT=yes
VLAN=yes
BOOTPROTO=dhcp
DEFROUTE=yes
NM_CONTROLLED=no
HOTPLUG=no

2) second issue seems to be that if dhclient is already running on the host, host fails to obtain address (see logs bellow)


logs:

MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,105::ifcfg::537::root::(writeConfFile) Writing to file /etc/sysconfig/network-scripts/ifcfg-em1.172 configuration:
# Generated by VDSM version 4.14.7-2.el6ev
DEVICE=em1.172
ONBOOT=yes
VLAN=yes
BOOTPROTO=dhcp
DEFROUTE=yes
NM_CONTROLLED=no

MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,169::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,171::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,175::utils::642::root::(execCmd) '/sbin/ip -d link show dev em1.172' (cwd None)
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,177::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,179::utils::642::root::(execCmd) '/sbin/ifup em1.172' (cwd None)
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,207::utils::662::root::(execCmd) FAILED: <err> = 'dhclient(7333) is already running - exiting. \n\nThis version of ISC DHCP is based on the release available\non ftp.isc.org.  Features have been added and other changes\nhave been made to the base software release in order to make\nit work better with this distribution.\n\nPlease report for this software via the Red Hat Bugzilla site:\n    http://bugzilla.redhat.com\n\nexiting.\n'; <rc> = 1
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,207::utils::642::root::(execCmd) '/sbin/ifdown em1.172' (cwd None)
sourceRoute::DEBUG::2014-05-21 13:13:16,315::sourceRouteThread::20::root::(process_IN_CLOSE_WRITE_filePath) Responding to DHCP response in /var/run/vdsm/sourceRoutes/1400670796
sourceRoute::INFO::2014-05-21 13:13:16,316::sourceRoute::175::root::(remove) Removing gateway - device: em1.172
sourceRoute::DEBUG::2014-05-21 13:13:16,316::utils::642::root::(execCmd) '/sbin/ip rule' (cwd None)
sourceRoute::DEBUG::2014-05-21 13:13:16,326::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
sourceRoute::ERROR::2014-05-21 13:13:16,326::sourceRoute::162::root::(_getRules) Routing rules not found for device em1.172
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,394::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-13::INFO::2014-05-21 13:13:16,395::ifcfg::339::root::(restoreAtomicNetworkBackup) Rolling back logical networks configuration (restoring atomic logical networks backup)
MainProcess|Thread-13::INFO::2014-05-21 13:13:16,395::ifcfg::379::root::(restoreAtomicBackup) Rolling back configuration (restoring atomic backup)
MainProcess|Thread-13::INFO::2014-05-21 13:13:16,395::ifcfg::388::root::(restoreAtomicBackup) Restored /etc/sysconfig/network-scripts/ifcfg-em1.172
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,395::utils::642::root::(execCmd) '/sbin/ifup em1.172' (cwd None)
MainProcess|Thread-13::DEBUG::2014-05-21 13:13:16,510::utils::662::root::(execCmd) FAILED: <err> = '/etc/sysconfig/network-scripts/ifup-eth: line 282:  9426 Terminated              /sbin/dhclient ${DHCLIENTARGS} ${DEVICE}\n'; <rc> = 1
MainProcess|Thread-13::ERROR::2014-05-21 13:13:16,511::ifcfg::431::root::(_startDevices) Failed to ifup device em1.172 during rollback.
Traceback (most recent call last):
  File "/usr/share/vdsm/netconf/ifcfg.py", line 428, in _startDevices
    ifup(dev)
  File "/usr/share/vdsm/netconf/ifcfg.py", line 789, in ifup
    rc, out, err = _ifup(iface)
  File "/usr/share/vdsm/netconf/ifcfg.py", line 778, in _ifup
    out[-1] if out else '')
ConfigNetworkError: (29, 'Determining IP information for em1.172... failed.')
MainProcess|Thread-13::ERROR::2014-05-21 13:13:16,511::supervdsmServer::100::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 98, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 202, in setupNetworks
    return configNetwork.setupNetworks(networks, bondings, **options)
  File "/usr/share/vdsm/configNetwork.py", line 651, in setupNetworks
    implicitBonding=True, **d)
  File "/usr/share/vdsm/configNetwork.py", line 186, in wrapped
    return func(*args, **kwargs)
  File "/usr/share/vdsm/configNetwork.py", line 288, in addNetwork
    netEnt.configure(**options)
  File "/usr/share/vdsm/netmodels.py", line 123, in configure
    self.configurator.configureVlan(self, **opts)
  File "/usr/share/vdsm/netconf/ifcfg.py", line 89, in configureVlan
    ifup(vlan.name, vlan.ipConfig.async)
  File "/usr/share/vdsm/netconf/ifcfg.py", line 789, in ifup
    rc, out, err = _ifup(iface)
  File "/usr/share/vdsm/netconf/ifcfg.py", line 778, in _ifup
    out[-1] if out else '')
ConfigNetworkError: (29, 'Determining IP information for em1.172... failed.')
sourceRoute::DEBUG::2014-05-21 13:13:16,522::sourceRouteThread::20::root::(process_IN_CLOSE_WRITE_filePath) Responding to DHCP response in /var/run/vdsm/sourceRoutes/1400670796
sourceRoute::INFO::2014-05-21 13:13:16,526::sourceRouteThread::37::root::(process_IN_CLOSE_WRITE_filePath) interface em1.172 is not a libvirt interface

Comment 1 Dan Kenigsberg 2014-05-22 22:40:24 UTC
This race is as old as vdsm networking, no reason for it to block rhev-3.4.0.

Comment 5 Martin Pavlik 2014-07-17 16:27:44 UTC
verified VDSM version 4.16.0-1.el6_5

Comment 6 Dan Kenigsberg 2014-07-28 12:27:49 UTC
Removing clone request, as the issue is already merged to upstream stable branch http://gerrit.ovirt.org/#/c/28438/

Comment 10 errata-xmlrpc 2015-02-11 21:11:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0159.html


Note You need to log in before you can comment on or make changes to this bug.