Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1514450 - [RFE] Configurable openshift logs to custom file location [NEEDINFO]
Summary: [RFE] Configurable openshift logs to custom file location
Keywords:
Status: NEW
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RFE
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Brenton Leanhardt
QA Contact: Xiaoli Tian
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-17 13:12 UTC by Bruno Andrade
Modified: 2019-03-30 07:07 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
bleanhar: needinfo? (rfoyle)


Attachments (Terms of Use)

Description Bruno Andrade 2017-11-17 13:12:45 UTC
1. Why exactly do you need this feature? (List the business requirements here, it's important to emphasize why secure-forwarder plugin is feasible for your case )

Openshift log information is directed by default to /var/log/messages. /var/log storage is assigned a default space of 3GB.
The Openshift messages are quickly filling up the /var/log folder. Hence Openshift team is using a customized log rotation
that runs thrice a day and gzips the messages file when rotated. Whereas Linux team wants to implement the default
Linux file rotation policy that runs once a day,keeps 30 copies  and do not do a gzip. Unix team also has concerns that the gzip of messages some time does not allow the log information from being retained in journal.  With default log rotation policy, Openshift node's /var/log/messages will fill up much more frequently and needs file system expansion to increase the /var/log size. Whereas Unix team have challenges to expand the file system of /var/log (being in root vg)\
Hence we need a configuration option to capture the Openshift logs in a custom file location


 2. How would you like to achieve this? (List the functional requirements here) . Below are the functional requirements at a high level. I will add more after speaking to the team (if any)

a. Need a configurable parameter in master config, that allows to direct the Openshift related logs of master nodes to a custom
    file location (Ex : /ocp/logs/openshift.log)

b. Need a configurable parameter in master config, that allows to direct the Openshift related logs of application nodes to a custom   file location (Ex : /ocp/logs/openshift.log)

c. Need a configurable parameter in master config, that allows to direct the Openshift related logs of etcd nodes to a custom file location (Ex : /ocp/logs/openshift.log)

d. Need to ensure that the custom Openshift logs of master nodes, application nodes and etcd nodes are captured in sos reports

e. Openshift related logs are to persisted in journal as it happens by default with the current configuration

f. Migrating these logs should not have any impacts on metrics & logging . All the application pod logs and infrastructure pod logs should be available in kibana as it works now in 
   the current configuration

g. Metrics functionality should be in intact after the log configuration for all application pods and infrastructure pods


3. Do you have any specific timeline dependencies? 
Our OCP Greenfield 3.6 Environments will go live in Feb 2018.  It is helpful to have this solution before Feb 2018. 


4. Would you be able to assist in testing this functionality if implemented? 
Yes. I am able to support to test the functionality

Comment 1 Rich Megginson 2017-11-17 14:51:48 UTC
(In reply to Bruno Andrade from comment #0)
> 1. Why exactly do you need this feature? (List the business requirements
> here, it's important to emphasize why secure-forwarder plugin is feasible
> for your case )
> 
> Openshift log information is directed by default to /var/log/messages.
> /var/log storage is assigned a default space of 3GB.

This is not strictly true.

OpenShift packaged as RPMs, and running as systemd services, either log to syslog(3), which by default go to journald, or log to stdout/stderr, which by default as a systemd service will also go to journald.  If journald is configured with Storage=Persistent, the logs go to /var/log/journald by default.  journald has a number of parameters which can help with disk space: Compress, SystemMaxFileSize, SystemMaxFiles, etc.

The next step is rsyslog - by default, rsyslog is configured to read logs from journald and write them to /var/log/messages.  If you do not need /var/log/messages, you can disable rsyslog from writing this file, and prevent log duplication.  IMO, /var/log/messages is not needed.

You might be able to achieve your goals by configuring journald and disabling rsyslog.

> The Openshift messages are quickly filling up the /var/log folder. Hence
> Openshift team is using a customized log rotation
> that runs thrice a day and gzips the messages file when rotated. Whereas
> Linux team wants to implement the default
> Linux file rotation policy that runs once a day,keeps 30 copies  and do not
> do a gzip. Unix team also has concerns that the gzip of messages some time
> does not allow the log information from being retained in journal.  With
> default log rotation policy, Openshift node's /var/log/messages will fill up
> much more frequently and needs file system expansion to increase the
> /var/log size. Whereas Unix team have challenges to expand the file system
> of /var/log (being in root vg)\
> Hence we need a configuration option to capture the Openshift logs in a
> custom file location
> 
> 
>  2. How would you like to achieve this? (List the functional requirements
> here) . Below are the functional requirements at a high level. I will add
> more after speaking to the team (if any)
> 
> a. Need a configurable parameter in master config, that allows to direct the
> Openshift related logs of master nodes to a custom
>     file location (Ex : /ocp/logs/openshift.log)
> 
> b. Need a configurable parameter in master config, that allows to direct the
> Openshift related logs of application nodes to a custom   file location (Ex
> : /ocp/logs/openshift.log)
> 
> c. Need a configurable parameter in master config, that allows to direct the
> Openshift related logs of etcd nodes to a custom file location (Ex :
> /ocp/logs/openshift.log)

Note that this means the OCP EFK logging stack will _not_ be able to find these logs.  In 3.6 and later, OCP EFK will only look for system/openshift logs in journald.

> 
> d. Need to ensure that the custom Openshift logs of master nodes,
> application nodes and etcd nodes are captured in sos reports
> 
> e. Openshift related logs are to persisted in journal as it happens by
> default with the current configuration
> 
> f. Migrating these logs should not have any impacts on metrics & logging .
> All the application pod logs and infrastructure pod logs should be available
> in kibana as it works now in 
>    the current configuration
> 
> g. Metrics functionality should be in intact after the log configuration for
> all application pods and infrastructure pods
> 
> 
> 3. Do you have any specific timeline dependencies? 
> Our OCP Greenfield 3.6 Environments will go live in Feb 2018.  It is helpful
> to have this solution before Feb 2018. 
> 
> 
> 4. Would you be able to assist in testing this functionality if implemented? 
> Yes. I am able to support to test the functionality

Comment 6 Richard Foyle 2018-03-27 19:23:45 UTC
The customer requirement is from an system team to remove Openshift logging from /var/log/messages and a way to filter out an aggregate (Splunk). Currently they are using remote syslog.

Comment 7 Rich Megginson 2018-03-27 19:54:53 UTC
(In reply to Richard Foyle from comment #6)
> The customer requirement is from an system team to remove Openshift logging
> from /var/log/messages

You can edit /etc/rsyslog.conf to stop it from writing to /var/log/messages.  Just comment out the line like this:

#*.info;mail.none;authpriv.none;cron.none                /var/log/messages

Openshift doesn't actually log to /var/log/messages - it logs to syslog which goes to the journal.  You can view openshift logs like this:

journalctl -u atomic-openshift-master
journalctl -u atomic-openshift-node

etc.

systemctl list-units | grep openshift

to see the list of openshift components

Starting with OCP 3.6, fluentd does not use /var/log/messages.  It will read system logs from journald.

> and a way to filter out an aggregate (Splunk).
> Currently they are using remote syslog.

Are they using https://docs.openshift.com/container-platform/3.7/install_config/aggregate_logging.html#sending-logs-to-external-rsyslog ?

Given the answers above, what does the customer want?
Do they want native splunk support so they don't have to use the remote-syslog plugin from fluentd?

Comment 8 Richard Foyle 2018-03-27 21:23:49 UTC
The customer expects that we will test a way , hence the RFE to accomplish the use case. The Implementation detail is up to Red Hat.

Comment 9 Rich Megginson 2018-03-27 23:24:57 UTC
(In reply to Richard Foyle from comment #8)
> The customer expects that we will test a way , hence the RFE to accomplish
> the use case. The Implementation detail is up to Red Hat.

Customer wants this?

When installing OpenShift using openshift-ansible, there should be a configuration parameter which turns off rsyslog writing to /var/log/messages.  This should be QE tested by OpenShift to verify that this introduces no regressions in the behavior of OpenShift or the operating system.  Customer should be aware that there may be other applications not under the control of RHEL or OpenShift that may not function correctly if /var/log/messages is not present.  All we can do is verify that RHEL and OpenShift continue to function correctly.  Customer should also understand that they will need to use the journalctl command to view logs from RHEL and OpenShift components since /var/log/messages will not be used.  Is that understood?

I think splunk integration is outside the scope of this bz.

Comment 10 Rich Megginson 2018-04-03 15:33:19 UTC
This isn't a logging issue.  This is an issue with origin or openshift-ansible.  Please reassign to someone who deals with those issues.  If/when openshift is changed to log to a different location, _then_ it will become a logging issue that we will have to deal with, by changing the log collector to read logs from alternate locations.

Comment 11 Richard Foyle 2018-04-11 14:35:27 UTC
Customer response to questions ia below. Please note that the original ask is that we have a way to filter out messages from /var/log/messages and that RH understand their use case and would have a QE test and/or update them if we design out the custom solution from working.


Response:
. They are asking in Greenfield environment if you use the following? Please confirm. https://docs.openshift.com/container-platform/3.7/install_config/aggregate_logging.html#sending-logs-to-external-rsyslog ? 

We do not use this feature of fluent-plugin-remote-syslog plug-in in OCP 3.6. We will test and implement this in future.

2. Now given the current version and way of handling logging is there still am ask (RFE)? 

We still have the requirement to migrate openshift logs out of /var/log/messages to another custom  file location . 

3. For instance, do you want native splunk support in OCP so that the remote-syslog plugin from fluentd is no longer needed?

I did not get the part on native splunk support. When we tested the migration of logs, the custom file location was /opt/splunk and  it did not involve any splunk setup.

Comment 12 Rich Megginson 2018-04-11 14:38:26 UTC
I think this is an origin issue - the ask is to configure the various origin components (master api, node, etcd, others?) to log to files in custom locations.

Once that is done, there will be follow on issue for OpenShift Logging to be able to read and handle these files.


Note You need to log in before you can comment on or make changes to this bug.