Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1366367 - Heapster "Failed to find node" warning and verbose logging
Summary: Heapster "Failed to find node" warning and verbose logging
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.2.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Matt Wringe
QA Contact: chunchen
URL:
Whiteboard:
Depends On: 1366365
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-11 18:47 UTC by Matt Wringe
Modified: 2016-09-30 02:17 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, heapster incorrectly generated error messages indicating that it "Failed to find node". We have corrected this error ensuring that erroneous warnings are generated.
Clone Of: 1366365
Environment:
Last Closed: 2016-09-12 17:36:14 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:1853 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 3.2 security update and bug fix update 2016-09-12 21:33:16 UTC

Description Matt Wringe 2016-08-11 18:47:58 UTC
+++ This bug was initially created as a clone of Bug #1366365 +++

Description of problem:
When running Heapster, the logs will fill up with warning messages about "Failed to find node" as well as more verbose messages about each and every invocation to the nodes and how many metrics were returned.

The warning about "Failed to find node" is confusing and has nothing to do with Heapster being able to find or connect to OpenShift nodes.

These warning messages have been fixed upstream by changing the message and severity. See https://github.com/kubernetes/heapster/commit/c192564d3b551a2a632c067a9121fc6d239ebe24

We should backport this change into our images. 


Version-Release number of selected component (if applicable):
3.3.0, 3.2.1

How reproducible:
Always

Steps to Reproduce:
1. deploy metrics and view the Heapster logs

Actual results:
no warnings or errors in the logs unless a real error or issue has occured.

Comment 1 Troy Dawson 2016-08-11 19:55:48 UTC
This is fixed for openshift 3.2 in rpm heapster-1.1.0-1.beta2.el7.1
That rpm is in image openshift3/metrics-heapster:3.2.1-3 and newer.

Comment 3 chunchen 2016-08-12 04:56:16 UTC
It's fixed, checked with the latest metrics images, please refer to the below testing results:

<----------snip--------->
The endpoint check has successfully completed.
Starting Heapster with the following arguments: --source=kubernetes:https://ec2-52-90-81-201.compute-1.amazonaws.com:443?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=6qmPC5CqtnKs-2B&filter=label(container_name:^system.slice.*|^user.slice) --tls_cert=/secrets/heapster.cert --tls_key=/secrets/heapster.key --tls_client_ca=/secrets/heapster.client-ca --allowed_users=system:master-proxy --metric_resolution=10s
I0811 23:58:42.672825       1 heapster.go:65] heapster --source=kubernetes:https://ec2-52-90-81-201.compute-1.amazonaws.com:443?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=6qmPC5CqtnKs-2B&filter=label(container_name:^system.slice.*|^user.slice) --tls_cert=/secrets/heapster.cert --tls_key=/secrets/heapster.key --tls_client_ca=/secrets/heapster.client-ca --allowed_users=system:master-proxy --metric_resolution=10s
I0811 23:58:42.673766       1 heapster.go:66] Heapster version 1.1.0-beta2
I0811 23:58:42.674124       1 configs.go:60] Using Kubernetes client with master "https://ec2-52-90-81-201.compute-1.amazonaws.com:443" and version "v1"
I0811 23:58:42.674161       1 configs.go:61] Using kubelet port 10250
I0811 23:58:42.743252       1 driver.go:322] Initialised Hawkular Sink with parameters {_system https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=6qmPC5CqtnKs-2B&filter=label(container_name:^system.slice.*|^user.slice) 0xc20817b0e0  5}
I0811 23:58:43.443283       1 heapster.go:92] Starting with Hawkular-Metrics Sink
I0811 23:58:43.443306       1 heapster.go:92] Starting with Metric Sink
I0811 23:58:43.454821       1 heapster.go:171] Starting heapster on port 8082
W0812 00:47:56.641961       1 manager.go:119] Failed to push data to sink: Hawkular-Metrics Sink

Comment 5 errata-xmlrpc 2016-09-12 17:36:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1853


Note You need to log in before you can comment on or make changes to this bug.