Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1688968 - after upgrade rawhide "Virtual network 'default': NAT" became Inactive
Summary: after upgrade rawhide "Virtual network 'default': NAT" became Inactive
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-14 19:49 UTC by Mikhail
Modified: 2019-03-29 19:19 UTC (History)
11 users (show)

Fixed In Version: libvirt-5.1.0-3.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-29 19:19:07 UTC


Attachments (Terms of Use)
libvirtd.log (deleted)
2019-03-15 15:15 UTC, Mikhail
no flags Details
iptables-save and ip6tables-save (deleted)
2019-03-15 15:18 UTC, Mikhail
no flags Details

Description Mikhail 2019-03-14 19:49:35 UTC
Description of problem:
after upgrade rawhide "Virtual network 'default': NAT" became Inactive

# virsh net-list --all
 Name      State      Autostart   Persistent
----------------------------------------------
 default   inactive   yes         yes

# virsh net-start default
error: Failed to start network default
error: COMMAND_FAILED: '/usr/sbin/iptables -w10 -w --table filter --insert LIBVIRT_INP --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: No chain/target/match by that name.

# rpm -qa | grep libvirt | sort
libvirt-bash-completion-5.1.0-2.fc31.x86_64
libvirt-client-5.1.0-2.fc31.x86_64
libvirt-daemon-5.1.0-2.fc31.x86_64
libvirt-daemon-config-network-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-interface-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-network-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-nodedev-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-nwfilter-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-qemu-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-secret-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-core-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-disk-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-gluster-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-iscsi-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-iscsi-direct-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-logical-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-mpath-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-rbd-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-scsi-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-sheepdog-5.1.0-2.fc31.x86_64
libvirt-daemon-driver-storage-zfs-5.1.0-2.fc31.x86_64
libvirt-daemon-kvm-5.1.0-2.fc31.x86_64
libvirt-gconfig-2.0.0-3.fc30.x86_64
libvirt-glib-2.0.0-3.fc30.x86_64
libvirt-gobject-2.0.0-3.fc30.x86_64
libvirt-libs-5.1.0-2.fc31.x86_64
python3-libvirt-5.1.0-1.fc31.x86_64

Comment 1 Mikhail 2019-03-15 04:19:38 UTC
Last working libvirt:

# rpm -qa | grep libvirt | sort
libvirt-daemon-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-interface-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-network-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-nodedev-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-nwfilter-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-qemu-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-secret-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-core-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-disk-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-gluster-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-iscsi-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-iscsi-direct-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-logical-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-mpath-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-rbd-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-scsi-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-sheepdog-5.0.0-2.fc30.x86_64
libvirt-daemon-driver-storage-zfs-5.0.0-2.fc30.x86_64
libvirt-daemon-kvm-5.0.0-2.fc30.x86_64
libvirt-gconfig-2.0.0-3.fc30.x86_64
libvirt-glib-2.0.0-3.fc30.x86_64
libvirt-gobject-2.0.0-3.fc30.x86_64
libvirt-libs-5.0.0-2.fc30.x86_64
python3-libvirt-5.1.0-1.fc31.x86_64

Comment 2 Daniel Berrange 2019-03-15 09:41:34 UTC
Libvirt creates the reference iptables rule at startup. If it is missing then most likely one or more of the required iptables modules is missing. Usually this is seen on machines where the admin has blacklisted ipv6.

The current libvirt code requires the ip6tables modules as well as iptables modules, though we're likely to relax that requirement given how many people have hit this.

Comment 3 Mikhail 2019-03-15 09:51:36 UTC
> Usually this is seen on machines where the admin has blacklisted ipv6.
I did not blacklisted ipv6 on my machine.
I just updated libvirt from 5.0.0-2 to 5.1.0-2.

Comment 4 Daniel Berrange 2019-03-15 10:12:43 UTC
Interesting, can you collect some debug info

Edit /etc/libvirt/libvirtd.conf and set

    log_filters="1:firewall 1:network 1:iptables"
     log_outputs="1:file:/var/log/libvirt/libvirtd.log"

The

  systemctl start libvirtd


And attempt to start the network again

Then attach that logfile to this bugzilla. 

It would also be useful to get attachment with output from

   iptables-save
   ip6tables-save

Comment 5 Jan Pokorný [poki] 2019-03-15 13:26:17 UTC
# journalctl -b -u libvirtd --no-hostname
> Mar 15 14:15:16 systemd[1]: Starting Virtualization daemon...
> Mar 15 14:15:17 systemd[1]: Started Virtualization daemon.
> Mar 15 14:15:19 libvirtd[4699]: libvirt version: 5.0.0, package: 2.fc30 (Fedora Project, 2019-02-01-18:25:08, )
> [...]
> Mar 15 14:15:19 libvirtd[4699]: Unable to open /dev/net/tun, is tun module loaded?: No such file or directory
> Mar 15 14:15:19 libvirtd[4699]: Unable to open /dev/net/tun, is tun module loaded?: No such file or directory
> Mar 15 14:15:19 libvirtd[4699]: Unable to open /dev/net/tun, is tun module loaded?: No such file or directory
> Mar 15 14:15:19 libvirtd[4699]: Unable to open /dev/net/tun, is tun module loaded?: No such file or directory

# lsmod | grep tun
> [nothing]
# modprobe tun
# echo $?
> 0
# lsmod | grep tun
> tun                    61440  0

# systemctl restart libvirtd

-->

everything works now, "tun" kernel module needs to be loaded explicitly
now, it seems

# uname -r
> 5.1.0-0.rc0.git7.1.fc31.x86_64

Comment 6 Daniel Berrange 2019-03-15 13:27:42 UTC
That's a bit odd - I'm not sure why missing tun module would cause libvirtd to fail to setup firewall rules

Comment 7 Mikhail 2019-03-15 15:15:42 UTC
Created attachment 1544499 [details]
libvirtd.log

Comment 8 Mikhail 2019-03-15 15:18:24 UTC
Created attachment 1544501 [details]
iptables-save and ip6tables-save

Comment 9 Mikhail 2019-03-15 15:28:54 UTC
# ifconfig
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.68  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::f9a9:2a39:2f99:e003  prefixlen 64  scopeid 0x20<link>
        ether 4c:ed:fb:75:5b:ab  txqueuelen 1000  (Ethernet)
        RX packets 2936085  bytes 3785903251 (3.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 880090  bytes 123691587 (117.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfe400000-fe41ffff  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 119  bytes 6385 (6.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 119  bytes 6385 (6.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vpn0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1406
        inet 10.76.146.109  netmask 255.255.254.0  destination 10.76.146.109
        inet6 fe80::7230:2390:76f4:7ce2  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 77325  bytes 22338225 (21.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 99078  bytes 17852270 (17.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Comment 10 Daniel Berrange 2019-03-15 15:35:32 UTC
Thanks, so that shows two things

 - there are no ip6tables top level chains at all
 - firewalld is reporting that it doens't have ip6tables support

    2019-03-15 15:13:03.859+0000: 31773: info : virFirewallApplyRule:723 : Applying rule '/usr/sbin/ip6tables --table filter --list-rules'
    2019-03-15 15:13:03.862+0000: 31773: error : virFirewallDApplyRule:332 : COMMAND_FAILED: UNKNOWN_ERROR: 'ip6tables' backend does not exist

You say you didn't disable ipv6, but it none the less looks like something related to ipv6 is disabled.

It could be just general Fedora rawhide explodyness

Comment 11 Mikhail 2019-03-15 15:39:02 UTC
> You say you didn't disable ipv6, but it none the less looks like something related to ipv6 is disabled.
Yes I didn't disable ipv6, but my ISP not support ipv6 and didn't gives ipv6 address to my computer.

Comment 12 Daniel Berrange 2019-03-15 15:41:15 UTC
I think it could well be something that's changed in firewalld. eg I think they might have switched over to the NFT kernel backend and be relying on compat support for iptables/ip6tables.

I'm installing a new rawhide VM to debug this further myself.

Comment 13 Jan Pokorný [poki] 2019-03-15 17:28:54 UTC
re [comment 5]:

I thought I was upon something that's a universal solution for the
problem at hand, since I also originally suffered the problem with
available networks marked as "inactive".  Perhaps the surface is more
convoluted, and I was just lucky to help myself with said easy step.

$ rpm -q firewalld nftables iptables systemd
> firewalld-0.6.3-2.fc30.noarch
> nftables-0.9.0-4.fc30.x86_64
> iptables-1.8.0-5.fc30.x86_64
> systemd-241-2.gita09c170.fc31.x86_64

Comment 14 Daniel Berrange 2019-03-18 14:16:03 UTC
So far I can reproduce the problem with missing /dev/net/tun  device & tun kmod. This appears to be a regression in Fedora 31 - not sure if is intentional or not (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/RW4WZYLPIMZ6K4JP5VYIPOWXP7LIFV3I/)

I cannot reproduce the missing IPv6 support though. A plain install of F31 gets a link-local IPv6 address regardless of whether the upstream link supports it or not.

None the less it is possible for people to blacklist ipv6.ko so we'll need to cope with that better.

Comment 15 Daniel Berrange 2019-03-19 15:36:26 UTC
The following series is merged upstream & will need backporting to Fedora 30

https://www.redhat.com/archives/libvir-list/2019-March/msg01218.html

Comment 16 Daniel Berrange 2019-03-19 16:45:49 UTC
WHile doing that we should also grab this other network related fix

https://www.redhat.com/archives/libvir-list/2019-March/msg00881.html

Comment 17 Fedora Update System 2019-03-21 11:16:27 UTC
libvirt-5.1.0-3.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-f3e6d3e0a5

Comment 18 Fedora Update System 2019-03-21 19:12:45 UTC
libvirt-5.1.0-3.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f3e6d3e0a5

Comment 19 Fedora Update System 2019-03-29 19:19:07 UTC
libvirt-5.1.0-3.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.