Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1512028 - operations logs lost if no ops cluster and no @OUTPUT label
Summary: operations logs lost if no ops cluster and no @OUTPUT label
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.6.z
Assignee: Rich Megginson
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1511719
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-10 17:21 UTC by Rich Megginson
Modified: 2017-12-07 06:49 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: When upgrading to a new version of logging, if the deployment does not use a separate OPS logging cluster, and there is no @OUTPUT label in the fluent.conf, fluentd does not know how to route the operations logs. Consequence: The operations logs are not logged. User will see no new operations (.operations.*) logs after the upgrade. Fix: Change the fluentd configuration to correctly handle the case both with and without the @OUTPUT label, and for the OPS and non-OPS cases. Result: Operations logs flow uninterrupted after upgrade.
Clone Of: 1511719
Environment:
Last Closed: 2017-12-07 06:49:06 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3390 normal SHIPPED_LIVE OpenShift Container Platform 3.6, 3.5, and 3.4 images update 2017-12-07 11:46:04 UTC
Github openshift origin-aggregated-logging pull 777 None None None 2017-11-10 17:21:52 UTC

Description Rich Megginson 2017-11-10 17:21:52 UTC
+++ This bug was initially created as a clone of Bug #1511719 +++

Description of problem:
Deploy logging with no ops cluster and based on 3.6 or an earlier 3.7 pre-release which does not have the @OUTPUT label in fluent.conf

update to a recent fluentd image

no operations logs are stored

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Rich Megginson on 2017-11-10 12:20:45 EST ---

https://github.com/openshift/origin-aggregated-logging/commit/61d1196eeab4ac02b360b20f9338d3ab2b5d5eef

Comment 1 Rich Megginson 2017-11-11 00:34:28 UTC
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=623860
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.6-rhel-7-docker-candidate-56826-20171111002326
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.73
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.73-5
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6

Comment 3 Anping Li 2017-11-14 07:56:28 UTC
I can reproduced by deploy logging:3.6.173.0.49 using openshift-ansible-                       3.6.173.0.5.

Once we update to 3.6.173.0.63, no such issue.

logging-kibana/images/3.6.173.0.63-10
logging-elasticsearch/images/3.6.173.0.63-10
logging-auth-proxy/images/3.6.173.0.63-10
logging-fluentd/images/3.6.173.0.63-10
logging-curator/images/3.6.173.0.63-10

oc exec -c elasticsearch logging-es-data-master-foodc7mt-2-3bw56 -- curl -s -XGET --cacert /etc/elasticsearch/secret/admin-ca --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://localhost:9200/.operations*/_count
{"count":421,"_shards":{"total":1,"successful":1,"failed":0}}

Comment 6 errata-xmlrpc 2017-12-07 06:49:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3390


Note You need to log in before you can comment on or make changes to this bug.