Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1518507 - Messages log is spammed with 'COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables' warnings on VM shutdown
Summary: Messages log is spammed with 'COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F ...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Virt
Version: 4.2.0
Hardware: x86_64
OS: Linux
unspecified
medium vote
Target Milestone: ---
: ---
Assignee: Michal Skrivanek
QA Contact: meital avital
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-29 06:19 UTC by Michael Burman
Modified: 2017-12-01 19:44 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-01 19:44:36 UTC
oVirt Team: Network


Attachments (Terms of Use)
Logs (deleted)
2017-11-29 06:19 UTC, Michael Burman
no flags Details

Description Michael Burman 2017-11-29 06:19:08 UTC
Created attachment 1360169 [details]
Logs

Description of problem:
Messages log is spammed with 'COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables' warnings on VM shutdown.

The messages log is spammed with iptables warnings on VM shutdown.
Something trying to delete rules related to its tap device.
Possible race between firewalld and libvirt.

Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist.
Nov 29 08:01:21 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name.
Nov 29 08:01:22 camel-vdsa firewalld[837]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name.


Version-Release number of selected component (if applicable):
vdsm-4.20.8-53.gitc3edfc0.el7.centos.x86_64
4.2.0-0.0.master.20171127212333.git11e14b9.el7.centos
libvirt-client-3.2.0-14.el7_4.4.x86_64
kernel-3.10.0-693.11.1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Run VM on latest master
2. Shutdown the VM
3. Check messages log

Actual results:
Log is spammed with iptables warnings for the tap device

Comment 1 Dan Kenigsberg 2017-11-29 09:47:14 UTC
Laine, it seems like a race between libvirt and firewalld. Are you familiar with it? It seems unrelated to oVirt.

Comment 2 Dan Kenigsberg 2017-11-29 09:47:40 UTC
Burman, the errors are reported by firewalld[837]. Would you specify its version?

Comment 3 Michael Burman 2017-11-29 09:50:20 UTC
(In reply to Dan Kenigsberg from comment #2)
> Burman, the errors are reported by firewalld[837]. Would you specify its
> version?

firewalld-filesystem-0.4.4.4-6.el7.noarch
firewalld-0.4.4.4-6.el7.noarch

Comment 4 Laine Stump 2017-11-30 19:16:33 UTC
See Bug 1110880 (filed against the Fedora build of firewalld). In short, during cleanup libvirt issues commands to clear out rules that may or may not exist, and firewalld is reporting those as failures. In the past, they were logged as errors. firewalld's solution was to log them as warnings. At any rate, they are innocuous.

(Daniel's suggestion in the original bug was that firewalld should just return an error status without logging *anything*, since only the caller to firewalld knows whether or not a failure is an error. Unless they do this, the logs will continue to have the extra cruft)

Depending on your level of tolerance for innocuous warning logs, I guess you could either close this as NOTABUG, or reassign to firewalld asking them to stop logging altogether (I don't know if they would agree to that).

Comment 5 Dan Kenigsberg 2017-12-01 08:48:23 UTC
Thanks, Laine.

I understand that it is ugly to write, but would libvirt first check if a rule exists, and only later delete it?

Comment 6 Laine Stump 2017-12-01 18:51:19 UTC
Under normal circumstances that would double the number of API calls in order to remove rules, which could become significant when trying to start/stop a large number of guests in a short period (which is one of the reasons we switched from exec'ing firewall-cmd to using the dbus API directly, so we would lose a part of that gain that was apparently important to people concerned about scaling).

In the case of an error situation, where we're just trying to clean up after ourselves as expendiently as possible, it unnecessarily, and possibly disastrously, complicates the cleanup, potentially leading to a situation where cleaning up from one error creates another.

In the end, I think the more proper solution would be for firewalld to not log these warnings (as we initially requested) - what they're doing now is analogous to remove() logging a warning every time you call it to delete a file that doesn't exist - you can bet nobody would stand for that! :-)

Comment 7 Dan Kenigsberg 2017-12-01 19:44:36 UTC
I think we're stuck with these warnings, then.


Note You need to log in before you can comment on or make changes to this bug.