Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1512841 - VM didn't obtain IP after deleted tap interface and then restart network in host
Summary: VM didn't obtain IP after deleted tap interface and then restart network in host
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: rc
: ---
Assignee: Vlad Yasevich
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-14 09:30 UTC by jingzhao
Modified: 2017-11-20 14:21 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-20 14:21:06 UTC


Attachments (Terms of Use)
dmesg info of guest (deleted)
2017-11-14 09:33 UTC, jingzhao
no flags Details

Description jingzhao 2017-11-14 09:30:30 UTC
Description of problem:
VM didn't obtain IP after delete tap interface and then restart network in host

Version-Release number of selected component (if applicable):
[root@localhost ~]# uname -r
3.10.0-781.el7.x86_64
[root@localhost ~]# rpm -qa |grep qemu-kvm-rhev
qemu-kvm-rhev-2.10.0-5.el7.x86_64


How reproducible:
3/3

Steps to Reproduce:
1. Boot guest with qemu command line [1]

2. check the status of network in guest 
 # ping host ip and successfully

3. In host, delete tap interface and then reboot network
# ip link delete switch
# /etc/init.d/network restart

4. In guest, check the network status, ping time out

5. reboot guest and check the network
# dhclient
didn't obtain IP address

Actual results:
Didn't obtain IP address in guest


Expected results:
obtain IP address and ping successfully

Additional info:
1. restart network in guest status
[root@localhost ~]# /etc/init.d/network restart
Restarting network (via systemctl):  Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details.
                                                           [FAILED]
[root@localhost ~]# systemctl status network.service
● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-11-14 17:05:55 CST; 2min 46s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2697 ExecStop=/etc/rc.d/init.d/network stop (code=exited, status=0/SUCCESS)
  Process: 2887 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=1/FAILURE)

Nov 14 17:05:09 localhost.localdomain systemd[1]: Starting LSB: Bring up/down networking...
Nov 14 17:05:10 localhost.localdomain network[2887]: Bringing up loopback interface:  [  OK  ]
Nov 14 17:05:55 localhost.localdomain network[2887]: Bringing up interface eth0:  Error: Connection activa...c.)
Nov 14 17:05:55 localhost.localdomain network[2887]: [FAILED]
Nov 14 17:05:55 localhost.localdomain systemd[1]: network.service: control process exited, code=exited status=1
Nov 14 17:05:55 localhost.localdomain systemd[1]: Failed to start LSB: Bring up/down networking.
Nov 14 17:05:55 localhost.localdomain systemd[1]: Unit network.service entered failed state.
Nov 14 17:05:55 localhost.localdomain systemd[1]: network.service failed.


2. tcpdump info in guest
[root@localhost ~]# tcpdump -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:11:59.821013 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:12:08.164723 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:12:10.693205 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:12:10.724515 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:12:11.063012 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:12:11.333195 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:12:11.679259 IP6 :: > ff02::1:ff6c:6d6e: ICMP6, neighbor solicitation, who has localhost.localdomain, length 24
17:12:12.681254 IP6 localhost.localdomain > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:12:12.681995 IP6 localhost.localdomain > ff02::2: ICMP6, router solicitation, length 8
17:12:13.139244 IP6 localhost.localdomain > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:12:16.649482 IP6 localhost.localdomain > ff02::2: ICMP6, router solicitation, length 8
17:12:17.271715 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:12:20.652655 IP6 localhost.localdomain > ff02::2: ICMP6, router solicitation, length 8
17:12:35.487835 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:12:41.137505 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:12:50.679664 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:12:55.693198 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:12:55.721470 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:12:56.461293 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:12:56.509292 IP6 :: > ff02::1:ff6c:6d6e: ICMP6, neighbor solicitation, who has localhost.localdomain, length 24
17:12:57.511296 IP6 localhost.localdomain > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:12:57.648600 IP6 localhost.localdomain > ff02::2: ICMP6, router solicitation, length 8
17:12:57.668196 IP6 localhost.localdomain > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
17:13:01.649894 IP6 localhost.localdomain > ff02::2: ICMP6, router solicitation, length 8
17:13:02.204113 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:13:04.648588 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300
17:13:05.649959 IP6 localhost.localdomain > ff02::2: ICMP6, router solicitation, length 8
17:13:12.181950 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9a:6a:6b:6c:6d:6e (oui Unknown), length 300


3.NetworkMangaer status in guest
ot@localhost ~]# service NetworkManager status
Redirecting to /bin/systemctl status NetworkManager.service
● NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2017-11-14 17:14:37 CST; 1s ago
     Docs: man:NetworkManager(8)
  Process: 975 ExecStart=/usr/sbin/NetworkManager --no-daemon (code=exited, status=0/SUCCESS)
 Main PID: 975 (code=exited, status=0/SUCCESS)

Nov 14 17:13:27 localhost.localdomain NetworkManager[975]: <info>  [1510650807.6711] device (eth0): state ...d')
Nov 14 17:13:27 localhost.localdomain NetworkManager[975]: <info>  [1510650807.6713] manager: NetworkManag...TED
Nov 14 17:13:27 localhost.localdomain NetworkManager[975]: <info>  [1510650807.6714] policy: disabling aut...0'.
Nov 14 17:13:27 localhost.localdomain NetworkManager[975]: <warn>  [1510650807.6715] device (eth0): Activa...h0'
Nov 14 17:13:27 localhost.localdomain NetworkManager[975]: <info>  [1510650807.6717] device (eth0): state ...d')
Nov 14 17:13:27 localhost.localdomain NetworkManager[975]: <warn>  [1510650807.6857] firewall: [0x56131f8b...ED)
Nov 14 17:14:37 localhost.localdomain systemd[1]: Stopping Network Manager...
Nov 14 17:14:37 localhost.localdomain NetworkManager[975]: <info>  [1510650877.4810] caught SIGTERM, shutt...ly.
Nov 14 17:14:37 localhost.localdomain NetworkManager[975]: <info>  [1510650877.5055] exiting (success)
Nov 14 17:14:37 localhost.localdomain systemd[1]: Stopped Network Manager.


[1] qemu command line
/usr/libexec/qemu-kvm \
-M pc \
-cpu Haswell-noTSX \
-nodefaults -rtc base=utc \
-m 4G \
-smp 4,sockets=4,cores=1,threads=1 \
-enable-kvm \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-k en-us \
-nodefaults \
-global isa-debugcon.iobase=0x402 \
-boot menu=on \
-qmp tcp:0:6667,server,nowait \
-usb \
-device usb-tablet \
-vga qxl \
-drive file=/home/test/rhel75-64-virtio.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \
-device virtio-net-pci,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \
-monitor stdio \
-vnc :1 \

Comment 2 jingzhao 2017-11-14 09:33:14 UTC
Created attachment 1351843 [details]
dmesg info of guest

Comment 3 Vlad Yasevich 2017-11-14 14:26:21 UTC
I have a few clarification questions.

(In reply to jingzhao from comment #0)
> 
> Steps to Reproduce:
> 1. Boot guest with qemu command line [1]
> 
> 2. check the status of network in guest 
>  # ping host ip and successfully
> 
> 3. In host, delete tap interface and then reboot network
> # ip link delete switch
> # /etc/init.d/network restart

Is the 'switch' link a bridge or the tap device?  Deleting the bridge
will not delete links that used to be bridge ports.  It will just
removed them from the bridge.

Re-creating the bridge will only add bridge ports specified in NM
configuration.  Ports added manually (via qemu interface helpers or
by the admin) will not be added back.

If 'switch' is the tap device itself, deleting it while qemu is running
will make network fail completely without any ability to restore it.

> 
> [1] qemu command line
> /usr/libexec/qemu-kvm \
> -M pc \
> -cpu Haswell-noTSX \
> -nodefaults -rtc base=utc \
> -m 4G \
> -smp 4,sockets=4,cores=1,threads=1 \
> -enable-kvm \
> -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
> -k en-us \
> -nodefaults \
> -global isa-debugcon.iobase=0x402 \
> -boot menu=on \
> -qmp tcp:0:6667,server,nowait \
> -usb \
> -device usb-tablet \
> -vga qxl \
> -drive
> file=/home/test/rhel75-64-virtio.qcow2,if=none,id=drive-virtio-disk0,
> format=qcow2,cache=none,werror=stop,rerror=stop \
> -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \
> -device virtio-net-pci,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev
> tap,id=tap10 \
> -monitor stdio \
> -vnc :1 \

Can you add your /etc/qemu-ifup to this bz, so I can understand what's getting configured.

Thanks
Vlad

Comment 4 jingzhao 2017-11-15 02:14:26 UTC
(In reply to Vlad Yasevich from comment #3)
> I have a few clarification questions.
> 
> (In reply to jingzhao from comment #0)
> > 
> > Steps to Reproduce:
> > 1. Boot guest with qemu command line [1]
> > 
> > 2. check the status of network in guest 
> >  # ping host ip and successfully
> > 
> > 3. In host, delete tap interface and then reboot network
> > # ip link delete switch
> > # /etc/init.d/network restart
> 
> Is the 'switch' link a bridge or the tap device?  Deleting the bridge
> will not delete links that used to be bridge ports.  It will just
> removed them from the bridge.
> 
> Re-creating the bridge will only add bridge ports specified in NM
> configuration.  Ports added manually (via qemu interface helpers or
> by the admin) will not be added back.
> 
> If 'switch' is the tap device itself, deleting it while qemu is running
> will make network fail completely without any ability to restore it.
> 
> > 
> > [1] qemu command line
> > /usr/libexec/qemu-kvm \
> > -M pc \
> > -cpu Haswell-noTSX \
> > -nodefaults -rtc base=utc \
> > -m 4G \
> > -smp 4,sockets=4,cores=1,threads=1 \
> > -enable-kvm \
> > -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
> > -k en-us \
> > -nodefaults \
> > -global isa-debugcon.iobase=0x402 \
> > -boot menu=on \
> > -qmp tcp:0:6667,server,nowait \
> > -usb \
> > -device usb-tablet \
> > -vga qxl \
> > -drive
> > file=/home/test/rhel75-64-virtio.qcow2,if=none,id=drive-virtio-disk0,
> > format=qcow2,cache=none,werror=stop,rerror=stop \
> > -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \
> > -device virtio-net-pci,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev
> > tap,id=tap10 \
> > -monitor stdio \
> > -vnc :1 \
> 
> Can you add your /etc/qemu-ifup to this bz, so I can understand what's
> getting configured.

Host network info:

[root@localhost test]# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:23:24:80:99:d3  txqueuelen 1000  (Ethernet)
        RX packets 4001406  bytes 3458432899 (3.2 GiB)
        RX errors 0  dropped 2409  overruns 0  frame 0
        TX packets 4073611  bytes 4086921577 (3.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xf7c00000-f7c20000  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 27368  bytes 71390555 (68.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27368  bytes 71390555 (68.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

switch: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.66.4.211  netmask 255.255.252.0  broadcast 10.66.7.255
        inet6 fe80::223:24ff:fe80:99d3  prefixlen 64  scopeid 0x20<link>
        ether 00:23:24:80:99:d3  txqueuelen 1000  (Ethernet)
        RX packets 2747303  bytes 2904330084 (2.7 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1427792  bytes 3892260958 (3.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:f7:57:84  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


[root@localhost test]# cat /etc/qemu-ifup 
#!/bin/sh
switch=switch
/sbin/ifconfig $1 0.0.0.0 up
/usr/sbin/brctl addif ${switch} $1
/usr/sbin/brctl setfd ${switch} 0
/usr/sbin/brctl stp ${switch} off

Thanks
Jing

> 
> Thanks
> Vlad

Comment 5 Vlad Yasevich 2017-11-15 16:42:19 UTC
Based on the above information,  you are actually deleting the bridge that
was created to connect the the VM, not the tap device.

Is NetworkManager configured to create this bridge?

Additionally, after it's been removed and the network is restarted,  what is the state of that bridge? 

The reason I ask is that the only thing that would add the VM connected tap device to that bridge is /etc/qemu-ifup and that script is only executed when the VM originally starts.

So after removing the bridge and restarting the network, if the bridge is recreated via NM,  it still would not contain the tap port.  Thus the VM would have no connection to the network and would not be able to obtain the adress.

Have you tried added the tap device back to bridge to see if the network connectivity is restored?

Thanks
-vlad

Comment 6 jingzhao 2017-11-16 07:44:21 UTC
(In reply to Vlad Yasevich from comment #5)
> Based on the above information,  you are actually deleting the bridge that
> was created to connect the the VM, not the tap device.
> 
> Is NetworkManager configured to create this bridge?

--No, had stop the NetworkManager service 

> 
> Additionally, after it's been removed and the network is restarted,  what is
> the state of that bridge? 
> 
> The reason I ask is that the only thing that would add the VM connected tap
> device to that bridge is /etc/qemu-ifup and that script is only executed
> when the VM originally starts.
> 
> So after removing the bridge and restarting the network, if the bridge is
> recreated via NM,  it still would not contain the tap port.  Thus the VM
> would have no connection to the network and would not be able to obtain the
> adress.
> 
> Have you tried added the tap device back to bridge to see if the network
> connectivity is restored?

1. ip link delete switch 
2. # brctl show
bridge name	bridge id		STP enabled	interfaces
switch		8000.0023248099d3	no		eno1
virbr0		8000.525400f75784	yes		virbr0-nic
3. VM didn't obtain IP
4. # brctl show
bridge name	bridge id		STP enabled	interfaces
switch		8000.0023248099d3	no		eno1
							tap0
virbr0		8000.525400f75784	yes		virbr0-nic
5. After that, VM can get IP address

In host, we must add tap0 manually, after that, vm can get IP address

Thanks
Jing



> 
> Thanks
> -vlad

Comment 7 jingzhao 2017-11-16 07:45:38 UTC
(In reply to jingzhao from comment #6)
> (In reply to Vlad Yasevich from comment #5)
> > Based on the above information,  you are actually deleting the bridge that
> > was created to connect the the VM, not the tap device.
> > 
> > Is NetworkManager configured to create this bridge?
> 
> --No, had stop the NetworkManager service 
> 
> > 
> > Additionally, after it's been removed and the network is restarted,  what is
> > the state of that bridge? 
> > 
> > The reason I ask is that the only thing that would add the VM connected tap
> > device to that bridge is /etc/qemu-ifup and that script is only executed
> > when the VM originally starts.
> > 
> > So after removing the bridge and restarting the network, if the bridge is
> > recreated via NM,  it still would not contain the tap port.  Thus the VM
> > would have no connection to the network and would not be able to obtain the
> > adress.
> > 
> > Have you tried added the tap device back to bridge to see if the network
> > connectivity is restored?
> 
> 1. ip link delete switch 
> 2. # brctl show
> bridge name	bridge id		STP enabled	interfaces
> switch		8000.0023248099d3	no		eno1
> virbr0		8000.525400f75784	yes		virbr0-nic
> 3. VM didn't obtain IP
need add tap0 manually, # brctl addif switch tap0
> 4. # brctl show
> bridge name	bridge id		STP enabled	interfaces
> switch		8000.0023248099d3	no		eno1
> 							tap0
> virbr0		8000.525400f75784	yes		virbr0-nic
> 5. After that, VM can get IP address
> 
> In host, we must add tap0 manually, after that, vm can get IP address
> 
> Thanks
> Jing
> 
> 
> 
> > 
> > Thanks
> > -vlad

Comment 8 Vlad Yasevich 2017-11-20 14:21:06 UTC
So this is working as expected.  This doesn't appear to be a bug.  If you think otherwise,  feel free to re-open with justification.

vlad


Note You need to log in before you can comment on or make changes to this bug.