Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1518587 - Supervdsm log - ERROR - 'Failed source route addition:' on every dhclient lease renew
Summary: Supervdsm log - ERROR - 'Failed source route addition:' on every dhclient lea...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: 4.20.8
Hardware: x86_64
OS: Linux
medium
low vote
Target Milestone: ovirt-4.2.1
: ---
Assignee: Edward Haas
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-29 09:26 UTC by Michael Burman
Modified: 2018-02-12 11:54 UTC (History)
4 users (show)

Fixed In Version: v4.20.9-9-gf10278730
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-12 11:54:02 UTC
oVirt Team: Network
rule-engine: ovirt-4.2+


Attachments (Terms of Use)
supervdsm log (deleted)
2017-11-29 09:29 UTC, Michael Burman
no flags Details
supervdsm after fix (deleted)
2017-12-10 09:02 UTC, Michael Burman
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 85000 master MERGED net: Produce less noise when adding sourceroute 2017-12-06 15:57:28 UTC

Description Michael Burman 2017-11-29 09:26:56 UTC
Description of problem:
Supervdsm log - ERROR - 'Failed source route addition:' on every dhclient lease renew  

Each new dhclient lease, error is shown is the supervdsm log - 

MainProcess|dhclient-monitor::DEBUG::2017-11-29 09:39:56,213::sourceroute::190::root::(add) Adding source route for device ovirtmgmt
MainProcess|dhclient-monitor::DEBUG::2017-11-29 09:39:56,213::cmdutils::150::root::(exec_cmd) /sbin/ip -4 route add 0.0.0.0/0 via 10.35.128.254 dev ovirtmgmt table 170098706 (cwd None)
MainProcess|dhclient-monitor::DEBUG::2017-11-29 09:39:56,221::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'RTNETLINK answers: File exists\n'; <rc> = 2
MainProcess|dhclient-monitor::ERROR::2017-11-29 09:39:56,221::sourceroute::199::root::(add) Failed source route addition: ("IPRouteData(to='0.0.0.0/0' via='10.35.128.254' src=None family=4 device='ovirtmgmt' table='170098706')", 'RTNETLINK answers: File exists')


Version-Release number of selected component (if applicable):
vdsm-4.20.8-53.gitc3edfc0.el7.centos.x86_64

How reproducible:
100%
every new dhclient lease(every 4-5 hours)

Steps to Reproduce:
1. Run host for few hours and check the supervdsm log

Actual results:
Error each new lease

Comment 1 Michael Burman 2017-11-29 09:29:48 UTC
Created attachment 1360241 [details]
supervdsm log

Comment 2 Michael Burman 2017-12-10 09:00:28 UTC
Still see those on vdsm-4.20.9-25.git5bd5e94.el7.centos.x86_64 

MainProcess|dhclient-monitor::DEBUG::2017-12-10 05:24:13,353::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'RTNETLINK answers: File exists\n'; <rc> = 2
MainProcess|dhclient-monitor::DEBUG::2017-12-10 05:24:13,353::sourceroute::196::root::(add) Route already exists, addition failed,: ("IPRouteData(to='0.0.0.0/0' via='10.35.128.x' src=None family=4 device='ovirtmgmt' table='170098706')", 'RTNETLINK answers: File exists')

Comment 3 Michael Burman 2017-12-10 09:02:04 UTC
Created attachment 1365514 [details]
supervdsm after fix

Comment 4 Edward Haas 2017-12-10 10:11:39 UTC
(In reply to Michael Burman from comment #2)
> Still see those on vdsm-4.20.9-25.git5bd5e94.el7.centos.x86_64 
> 
> MainProcess|dhclient-monitor::DEBUG::2017-12-10
> 05:24:13,353::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'RTNETLINK
> answers: File exists\n'; <rc> = 2
> MainProcess|dhclient-monitor::DEBUG::2017-12-10
> 05:24:13,353::sourceroute::196::root::(add) Route already exists, addition
> failed,: ("IPRouteData(to='0.0.0.0/0' via='10.35.128.x' src=None family=4
> device='ovirtmgmt' table='170098706')", 'RTNETLINK answers: File exists')

It is now a debug message (was an error) and it specifies why it failed ("Route already exists").
This is the intended behaviour.

Comment 5 Michael Burman 2017-12-10 10:16:54 UTC
(In reply to Edward Haas from comment #4)
> (In reply to Michael Burman from comment #2)
> > Still see those on vdsm-4.20.9-25.git5bd5e94.el7.centos.x86_64 
> > 
> > MainProcess|dhclient-monitor::DEBUG::2017-12-10
> > 05:24:13,353::cmdutils::158::root::(exec_cmd) FAILED: <err> = 'RTNETLINK
> > answers: File exists\n'; <rc> = 2
> > MainProcess|dhclient-monitor::DEBUG::2017-12-10
> > 05:24:13,353::sourceroute::196::root::(add) Route already exists, addition
> > failed,: ("IPRouteData(to='0.0.0.0/0' via='10.35.128.x' src=None family=4
> > device='ovirtmgmt' table='170098706')", 'RTNETLINK answers: File exists')
> 
> It is now a debug message (was an error) and it specifies why it failed
> ("Route already exists").
> This is the intended behaviour.

I see, but 
1) It is spamming the log
2) I thought we going to remove this messages, because the route is exists, but dhclient trying to add new one and failing. It's going to spam the log on every new lease request from dhclient. is that expected?

Comment 6 Edward Haas 2017-12-10 10:33:44 UTC
(In reply to Michael Burman from comment #5)
> > It is now a debug message (was an error) and it specifies why it failed
> > ("Route already exists").
> > This is the intended behaviour.
> 
> I see, but 
> 1) It is spamming the log
> 2) I thought we going to remove this messages, because the route is exists,
> but dhclient trying to add new one and failing. It's going to spam the log
> on every new lease request from dhclient. is that expected?

We are not checking if something exists and only then add it, that is not safe (between the questioning and the adding the status can change).
Therefore, we always add and if it fails we check the error and report.

I think that the debug message is needed, because we may find ourself in cases where we expect one thing but something else is happening (like a wrong route set).
As this is occurring once per lease, I think it is fine. In most setups the lease is between a few hours to a few days.

Comment 7 Michael Burman 2017-12-10 11:19:51 UTC
I see, fine with me.

Comment 8 Michael Burman 2017-12-10 11:22:02 UTC
Base on comment6#, this bug is verified on - vdsm-4.20.9-27.gitb8f880e.el7.centos.x86_64

Comment 9 Sandro Bonazzola 2018-02-12 11:54:02 UTC
This bugzilla is included in oVirt 4.2.1 release, published on Feb 12th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.