Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1367349 - F24 kernel kvm bridge virtualization with NetworkManager is broken
Summary: F24 kernel kvm bridge virtualization with NetworkManager is broken
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 24
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-16 09:06 UTC by frantisek.reznicek
Modified: 2016-09-05 07:23 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-05 07:23:03 UTC


Attachments (Terms of Use)
The working configuration from F22 (deleted)
2016-08-16 09:08 UTC, frantisek.reznicek
no flags Details
F24 dmesg log (deleted)
2016-08-16 09:09 UTC, frantisek.reznicek
no flags Details
Excersize4, vnet0 tcpdump (deleted)
2016-08-16 09:32 UTC, frantisek.reznicek
no flags Details
Excersize3, vnet0 tcpdump (deleted)
2016-08-16 09:32 UTC, frantisek.reznicek
no flags Details
Excersize3, mbridge tcpdump (deleted)
2016-08-16 09:33 UTC, frantisek.reznicek
no flags Details
Excersize4, mbridge tcpdump (deleted)
2016-08-16 09:33 UTC, frantisek.reznicek
no flags Details

Description frantisek.reznicek 2016-08-16 09:06:49 UTC
Description of problem:
F24 kernel kvm bridge virtualization with NetworkManager is broken.

I'm currently unable to make KVM virtualization woring on a Laptop Dell Latitude E7470 with fresh installed F24 (KDE Plasma), NetworkManager enabled.

I configured bridging myself away from any GUI just from console.
My main bridge is named mbridge, bridging ethernet NIC enp0s31f6 and on demand KVM NICS vnet0 ...


The configuration looks following way (state when KVM VM is asking for IP using DHCP):
[root@marek-kaleta-dell ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.0242bad8234d       no
mbridge         8000.847beb48147c       yes             enp0s31f6
                                                        vnet0
virbr0          8000.000000000000       yes

[root@marek-kaleta-dell ~]# brctl showstp mbridge
mbridge
 bridge id              8000.847beb48147c
 designated root        8000.847beb48147c
 root port                 0                    path cost                  0
 max age                  20.00                 bridge max age            20.00
 hello time                2.00                 bridge hello time          2.00
 forward delay             2.00                 bridge forward delay       2.00
 ageing time             300.00
 hello timer               0.00                 tcn timer                  0.00
 topology change timer     0.00                 gc timer                  48.22
 flags


enp0s31f6 (1)                                                                                                                                                                                                               
 port id                8001                    state                forwarding                                                                                                                                             
 designated root        8000.847beb48147c       path cost                100                                                                                                                                                
 designated bridge      8000.847beb48147c       message age timer          0.00                                                                                                                                             
 designated port        8001                    forward delay timer        0.00                                                                                                                                             
 designated cost           0                    hold timer                 0.00                                                                                                                                             
 flags                                                                                                                                                                                                                      
                                                                                                                                                                                                                            
vnet0 (2)                                                                                                                                                                                                                   
 port id                8002                    state                forwarding                                                                                                                                             
 designated root        8000.847beb48147c       path cost                100                                                                                                                                                
 designated bridge      8000.847beb48147c       message age timer          0.00                                                                                                                                             
 designated port        8002                    forward delay timer        0.00                                                                                                                                             
 designated cost           0                    hold timer                 0.00                                                                                                                                             
 flags                                                                                                                                                                                                                      
It is suspicious that local mac addresses are doubled:
[root@marek-kaleta-dell ~]# brctl showmacs mbridge                                                                                                                                                                          
port no mac addr                is local?       ageing timer                                                                                                                                                                
  1     00:13:3b:99:9a:6e       no               173.52
  1     00:13:3b:99:db:5c       no                16.74
  1     00:13:3b:99:dd:4c       no                 3.31
  1     00:13:3b:9a:17:02       no                10.70
  1     00:13:3b:9a:17:4c       no                 3.14
  1     00:13:3b:9b:58:5f       no                20.55
  1     00:13:3b:9b:94:e2       no                12.98
  1     00:15:5d:88:fe:05       no                59.39
  1     00:15:5d:88:fe:23       no                28.61
  1     00:25:90:09:5a:0b       no                15.71
  1     00:41:d2:2c:55:83       no                 0.51
  1     08:00:27:5e:36:54       no                 1.43
  1     20:c9:d0:2b:bc:1d       no                29.06
  1     24:b6:fd:f8:10:b9       no                 0.10
  1     34:e6:d7:1a:75:cd       no                54.73
  1     34:e6:d7:33:21:6b       no                24.83
  1     34:e6:d7:33:23:f2       no                 1.55
  1     34:e6:d7:33:23:f3       no                40.82
  1     34:e6:d7:33:24:37       no                 0.54
  1     34:e6:d7:33:24:41       no                78.22
  1     34:e6:d7:33:24:98       no                78.22
  1     34:e6:d7:33:25:b8       no               272.23
  1     34:e6:d7:33:26:32       no                31.24
  1     34:e6:d7:33:27:0c       no                27.27
  2     52:54:00:79:0d:ca       no                15.69
  1     70:11:24:8c:5d:24       no                11.82
  1     70:11:24:8c:d5:d2       no                79.98
  1     70:11:24:8c:d8:ec       no                57.15
  1     74:86:7a:6d:9b:d5       no                78.22
  1     80:49:71:10:ca:88       no                60.05
  1     80:49:71:10:ca:a2       no                73.11
  1     80:49:71:11:12:c5       no                79.74
  1     84:7b:eb:10:02:28       no                26.70
  1     84:7b:eb:2c:a6:8c       no                 2.14
  1     84:7b:eb:44:c3:0e       no                81.12
  1     84:7b:eb:48:13:5f       no                65.15
  1     84:7b:eb:48:13:7d       no                 5.83
  1     84:7b:eb:48:13:bd       no                77.19
  1     84:7b:eb:48:13:c1       no               173.95
  1     84:7b:eb:48:14:7c       yes                0.00
  1     84:7b:eb:48:14:7c       yes                0.00
  1     ac:7f:3e:e5:dd:a8       no                53.28
  1     ac:7f:3e:e6:2c:5d       no                47.04
  1     ac:7f:3e:e6:31:43       no                77.59
  1     b8:27:eb:b2:09:26       no                 0.53
  1     b8:88:e3:94:44:7a       no                 8.72
  1     b8:ca:3a:a2:2a:d3       no                10.10
  1     d4:be:d9:75:d9:08       no                22.92
  1     e4:8d:8c:d7:b3:a0       no                56.78
  1     ec:f4:bb:62:02:e6       no               273.75
  1     ec:f4:bb:62:02:f7       no                76.61
  1     ec:f4:bb:62:05:6e       no                78.22
  1     ec:f4:bb:70:a5:b2       no                15.90
  1     f0:1f:af:04:d0:a8       no                57.49
  1     f0:1f:af:11:8f:b4       no                10.74
  1     f0:1f:af:19:a2:f5       no                34.62
  1     f0:1f:af:1e:63:06       no               102.44
  1     f0:1f:af:31:ad:87       no                 4.36
  1     f0:1f:af:38:8e:06       no                 5.89
  1     f0:1f:af:3b:f5:f4       no                 5.04
  1     f0:1f:af:3e:d9:c4       no                43.84
  1     f0:1f:af:3e:da:35       no               208.71
  1     f0:1f:af:3e:da:39       no                16.86
  1     f0:1f:af:3e:dd:c3       no                 0.78
  1     f0:1f:af:3e:e2:69       no                78.22
  1     f0:1f:af:3e:e2:6e       no                27.40
  1     f0:1f:af:46:38:57       no                 5.51
  1     f0:1f:af:46:3b:bc       no                 9.95
  1     f0:1f:af:46:3d:1b       no                65.38
  1     f0:1f:af:64:c0:4d       no                78.22
  1     f0:1f:af:64:c3:3a       no               206.03
  1     f4:f9:51:f0:b6:3d       no               206.02
  1     f4:f9:51:f0:c9:3c       no                78.09
  1     f4:f9:51:f1:b7:c0       no                49.51
  1     f8:ca:b8:08:e6:64       no                 5.83
  1     f8:ca:b8:17:d5:3d       no                 4.89
  1     f8:ca:b8:17:d5:48       no                78.22
  1     f8:ca:b8:17:d7:f2       no                37.08
  1     f8:ca:b8:17:d7:fd       no                31.55
  1     f8:ca:b8:2c:24:f7       no               174.18
  1     f8:ca:b8:2c:25:14       no                47.98
  1     f8:ca:b8:2c:25:15       no                78.22
  1     f8:ca:b8:5e:df:ce       no                26.34
  1     f8:ca:b8:60:27:a7       no                14.64
  1     f8:ca:b8:60:2b:ad       no                 5.27
  1     f8:ca:b8:60:2c:76       no                 1.07
  1     f8:ca:b8:62:0a:39       no                19.94
  2     fe:54:00:79:0d:ca       yes                0.00
  2     fe:54:00:79:0d:ca       yes                0.00


Configuration of the bridge and eth NIC are following:
[root@marek-kaleta-dell ~]# cat /etc/sysconfig/network-scripts/ifcfg-mbridge 
DEVICE=mbridge
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=dhcp
IPV6INIT=yes
IPV6_AUTOCONF=no
DHCPV6=no
STP=yes
DEFROUTE=yes
NAME=mbridge
#PEERDNS=yes
#PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
#UUID=5f216c41-c615-e916-7aff-74defaff18ea
#BRIDGING_OPTS=priority=32768
UUID=5f216c41-c615-e916-7aff-74defaff18ea
DELAY=2
BRIDGING_OPTS=priority=32768
PEERDNS=yes
PEERROUTES=yes
[root@marek-kaleta-dell ~]# cat /etc/sysconfig/network-scripts/ifcfg-enp0s31f6 
HWADDR=84:7B:EB:48:14:7C
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp0s31f6
UUID=f5d04d0e-8bc4-3bbb-904c-919aeed68ede
ONBOOT=yes
AUTOCONNECT_PRIORITY=-999
BRIDGE=mbridge

[root@marek-kaleta-dell ~]# sysctl -a 2>/dev/null  | grep bridge-nf-
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0

[root@marek-kaleta-dell ~]# sysctl -a 2>/dev/null | grep ipv4 | grep forwarding
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.docker0.forwarding = 1
net.ipv4.conf.docker0.mc_forwarding = 0
net.ipv4.conf.enp0s31f6.forwarding = 1
net.ipv4.conf.enp0s31f6.mc_forwarding = 0
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.mbridge.forwarding = 1
net.ipv4.conf.mbridge.mc_forwarding = 0
net.ipv4.conf.virbr0.forwarding = 1
net.ipv4.conf.virbr0.mc_forwarding = 0
net.ipv4.conf.virbr0-nic.forwarding = 1
net.ipv4.conf.virbr0-nic.mc_forwarding = 0
net.ipv4.conf.vnet0.forwarding = 1
net.ipv4.conf.vnet0.mc_forwarding = 0
net.ipv4.conf.wlp1s0.forwarding = 1
net.ipv4.conf.wlp1s0.mc_forwarding = 0

[root@marek-kaleta-dell ~]# ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:ba:d8:23:4d  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s31f6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 84:7b:eb:48:14:7c  txqueuelen 1000  (Ethernet)
        RX packets 78569  bytes 64132511 (61.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 19617  bytes 3577999 (3.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xe1300000-e1320000  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 16  bytes 1416 (1.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1416 (1.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

mbridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.133.213  netmask 255.255.255.0  broadcast 10.0.133.255
        inet6 fe80::2bc:328f:f55f:4814  prefixlen 64  scopeid 0x20<link>
        ether 84:7b:eb:48:14:7c  txqueuelen 1000  (Ethernet)
        RX packets 50422  bytes 61096874 (58.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18927  bytes 3428285 (3.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 00:00:00:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0-nic: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 52:54:00:68:89:cd  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc54:ff:fe79:dca  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:79:0d:ca  txqueuelen 1000  (Ethernet)
        RX packets 261  bytes 54270 (52.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13723  bytes 1832151 (1.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp1s0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether b8:08:cf:32:78:ca  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@marek-kaleta-dell ~]# ebtables -t filter -L --Lc
Bridge table: filter

Bridge chain: INPUT, entries: 1, policy: ACCEPT
-j INPUT_direct, pcnt = 49982 -- bcnt = 60865627

Bridge chain: FORWARD, entries: 1, policy: ACCEPT
-j FORWARD_direct, pcnt = 22732 -- bcnt = 3459522

Bridge chain: OUTPUT, entries: 1, policy: ACCEPT
-j OUTPUT_direct, pcnt = 18604 -- bcnt = 3131175

Bridge chain: INPUT_direct, entries: 0, policy: RETURN

Bridge chain: OUTPUT_direct, entries: 0, policy: RETURN

Bridge chain: FORWARD_direct, entries: 0, policy: RETURN




I followed same steps on Fedora 22 earlier and F22 works as expected. All F24 kernels I tried refuse to work (see list of kernels below).


Version-Release number of selected component (if applicable):
F24 up-to-date:
[root@marek-kaleta-dell ~]# uname -a
Linux marek-kaleta-dell.********** 4.6.6-300.fc24.x86_64 #1 SMP Wed Aug 10 21:07:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@marek-kaleta-dell ~]# rpm -q kernel
kernel-4.6.4-301.fc24.x86_64
kernel-4.6.5-300.fc24.x86_64
kernel-4.6.6-300.fc24.x86_64


How reproducible:
100%

Steps to Reproduce:
1. configure network bridge mbridge(enp0s31f6) via reconfiguring /etc/sysconfig/network-scripts/ifcfg-{mbridge,enp0s31f6} see above
2. make sure below sysctl values are set
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
3. make sure ebtables are off

Actual results:
Bridged main ethernet NIC does not work properly while creation of KVM VM machine (centos6.8), there is no DHCP offer coming.

Expected results:
Bridged main ethernet NIC should work properly while creation of KVM VM machine (centos6.8) there should be DHCP offer coming.

Additional info:

The virtualization host has main ethernet NIC IP configured with DHCP and this is functional:

Aug 16 10:59:18 marek-kaleta-dell.brno.seznam.cz dhclient[1736]: DHCPREQUEST on mbridge to 10.0.133.1 port 67 (xid=0x82eb650)
Aug 16 10:59:18 marek-kaleta-dell.brno.seznam.cz dhclient[1736]: DHCPACK from 10.0.133.1 (xid=0x82eb650)

When trying to get IP address for vnet0 (libvirt dynamic VM NIC) I get no DHCPACK:

[root@marek-kaleta-dell ~]# dhclient -d vnet0
Internet Systems Consortium DHCP Client 4.3.4
Copyright 2004-2016 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/vnet0/fe:54:00:79:0d:ca
Sending on   LPF/vnet0/fe:54:00:79:0d:ca
Sending on   Socket/fallback
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 6 (xid=0x2f031064)
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 8 (xid=0x2f031064)
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 14 (xid=0x2f031064)
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 11 (xid=0x2f031064)
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 17 (xid=0x2f031064)
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 5 (xid=0x2f031064)
No DHCPOFFERS received.

Comment 1 frantisek.reznicek 2016-08-16 09:08:23 UTC
Created attachment 1191151 [details]
The working configuration from F22

Comment 2 frantisek.reznicek 2016-08-16 09:09:08 UTC
Created attachment 1191152 [details]
F24 dmesg log

Comment 3 frantisek.reznicek 2016-08-16 09:31:45 UTC
It looks that DHCP server is not returning DHCPOFFER/DHCPACK for bridged vnet0 while for bridged enp0s31f6 it is working well.

I've tried two excersizes with capturing the traffic:

Excersize 3: KVM VM install procedure (anakonda running dhclient itself)

No DHCPOFFER/DHCPACK captured on both mbridge and vnet0, see attached files (cap_mbridge_3.cap, cap_vnet0_3.cap)

Excersize 4: manual dhclient on virtualization host (executed as root)
[root@marek-kaleta-dell ~]# dhclient -d vnet0
Internet Systems Consortium DHCP Client 4.3.4
Copyright 2004-2016 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/vnet0/fe:54:00:f2:7b:09
Sending on   LPF/vnet0/fe:54:00:f2:7b:09
Sending on   Socket/fallback
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 8 (xid=0xee8a5a0f)
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 13 (xid=0xee8a5a0f)
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 20 (xid=0xee8a5a0f)
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 18 (xid=0xee8a5a0f)
DHCPDISCOVER on vnet0 to 255.255.255.255 port 67 interval 2 (xid=0xee8a5a0f)
No DHCPOFFERS received.
No working leases in persistent database - sleeping.



No DHCPOFFER/DHCPACK captured on both mbridge and vnet0, see attached files (cap_mbridge_4.cap, cap_vnet0_4.cap)


Capturing is done following way:
tcpdump -w /tmp/cap_vnet0_<excersize-id>.cap -s 0 -i vnet0 'portrange 60-69'
tcpdump -w /tmp/cap_mbridge_<excersize-id>.cap -s 0 -i mbridge 'portrange 60-69'



Selinux in Enforcing, no AVCs.


Please suggest what else to try to resolve the issue.

Comment 4 frantisek.reznicek 2016-08-16 09:32:32 UTC
Created attachment 1191172 [details]
Excersize4, vnet0 tcpdump

Comment 5 frantisek.reznicek 2016-08-16 09:32:57 UTC
Created attachment 1191173 [details]
Excersize3, vnet0 tcpdump

Comment 6 frantisek.reznicek 2016-08-16 09:33:24 UTC
Created attachment 1191174 [details]
Excersize3, mbridge tcpdump

Comment 7 frantisek.reznicek 2016-08-16 09:33:44 UTC
Created attachment 1191175 [details]
Excersize4, mbridge tcpdump

Comment 8 frantisek.reznicek 2016-08-16 11:22:04 UTC
I'd need some suggestion how to check whether bridge forwards a traffic (nono DHCP) both directions as it may be problem with DHCP server.

Comment 9 frantisek.reznicek 2016-09-05 07:23:03 UTC
It appears to be the DHCP server which does not work the supposed way. Closing this one as NOT-A-BUG.


Note You need to log in before you can comment on or make changes to this bug.