Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1513408 - Continuously error info "record cannot use elasticsearch index name type project_full" in fluentd pod logs if use system container docker. [NEEDINFO]
Summary: Continuously error info "record cannot use elasticsearch index name type proj...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 3.7.z
Assignee: Jeff Cantrill
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks: 1551870
TreeView+ depends on / blocked
 
Reported: 2017-11-15 11:13 UTC by Junqi Zhao
Modified: 2019-04-15 15:12 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Messages for which the unique namespace id could not be determined could not be properly indexed. Consequence: Messages could be lost and the error message appears in the logs Fix: Modify the cache algorithm to provide the necessary data or default the value to 'orphaned'. Result: The error message is resolved and messages are stored in an 'orphaned' index when a unique namespace id can not be determined.
Clone Of:
: 1551870 (view as bug list)
Environment:
Last Closed: 2019-03-06 20:39:20 UTC
jcantril: needinfo? (smunilla)
gferrazs: needinfo? (jcantril)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 898 None None None 2018-01-17 23:42:08 UTC
Github openshift origin-aggregated-logging pull 904 None None None 2018-01-17 23:49:38 UTC

Description Junqi Zhao 2017-11-15 11:13:31 UTC
Description of problem:
Use system container docker,there are continuously info "record cannot use elasticsearch index name type project_full" in fluentd pod logs. 

This issue is only happen with system container docker, since its configure file is /etc/docker/container-daemon.json rather than  /etc/docker/daemon.json or /etc/sysconfig/docker.

Trace:
2017-11-15 10:19:25 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {"message"=>"2017-11-15 10:19:24 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {\"message\"=>\"2017-11-15T10:06:38.903902551Z AUDIT: id=\\\"968849aa-d7a7-45f2-9243-5ece6804e21f\\\" ip=\\\"10.130.0.1\\\" method=\\\"GET\\\" user=\\\"system:anonymous\\\" groups=\\\"\\\\\\\"system:unauthenticated\\\\\\\"\\\" as=\\\"<self>\\\" asgroups=\\\"<lookup>\\\" namespace=\\\"<none>\\\" uri=\\\"/healthz\\\"\\n\", \"level\"=>\"info\", \"hostname\"=>\"localhost.localdomain\", \"pipeline_metadata\"=>{\"collector\"=>{\"ipaddr4\"=>\"10.130.0.16\", \"ipaddr6\"=>\"fe80::c432:91ff:fe2f:d303\", \"inputname\"=>\"fluent-plugin-systemd\", \"name\"=>\"fluentd\", \"received_at\"=>\"2017-11-15T10:19:24.305099+00:00\", \"version\"=>\"0.12.39 1.6.0\"}}, \"@timestamp\"=>\"2017-11-15T10:06:38.904126+00:00\"}\n", "level"=>"info", "hostname"=>"localhost.localdomain", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.130.0.16", "ipaddr6"=>"fe80::c432:91ff:fe2f:d303", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2017-11-15T10:19:25.492302+00:00", "version"=>"0.12.39 1.6.0"}}, "@timestamp"=>"2017-11-15T10:19:24.308025+00:00"}
2017-11-15 10:19:25 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {"message"=>"2017-11-15 10:19:24 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {\"message\"=>\"2017-11-15T10:06:48.906137559Z AUDIT: id=\\\"73f3c8fa-f5a2-4160-b62f-747398429f3a\\\" ip=\\\"10.130.0.1\\\" method=\\\"GET\\\" user=\\\"system:anonymous\\\" groups=\\\"\\\\\\\"system:unauthenticated\\\\\\\"\\\" as=\\\"<self>\\\" asgroups=\\\"<lookup>\\\" namespace=\\\"<none>\\\" uri=\\\"/healthz\\\"\\n\", \"level\"=>\"info\", \"hostname\"=>\"localhost.localdomain\", \"pipeline_metadata\"=>{\"collector\"=>{\"ipaddr4\"=>\"10.130.0.16\", \"ipaddr6\"=>\"fe80::c432:91ff:fe2f:d303\", \"inputname\"=>\"fluent-plugin-systemd\", \"name\"=>\"fluentd\", \"received_at\"=>\"2017-11-15T10:19:24.305285+00:00\", \"version\"=>\"0.12.39 1.6.0\"}}, \"@timestamp\"=>\"2017-11-15T10:06:48.906337+00:00\"}\n", "level"=>"info", "hostname"=>"localhost.localdomain", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.130.0.16", "ipaddr6"=>"fe80::c432:91ff:fe2f:d303", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2017-11-15T10:19:25.492472+00:00", "version"=>"0.12.39 1.6.0"}}, "@timestamp"=>"2017-11-15T10:19:24.308032+00:00"}

******************************************************************************


# docker info | grep 'Logging Driver'
Logging Driver: json-file

# cat /etc/docker/daemon.json 
{}

#cat /etc/sysconfig/docker
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'



Version-Release number of selected component (if applicable):
logging componet version is v3.7.8-1
# openshift version
openshift v3.7.8
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8


How reproducible:
Always

Steps to Reproduce:
1. Deploy logging 3.7


Actual results:
continuously info "record cannot use elasticsearch index name type project_full" in fluentd pod logs

Expected results:
Should not be error in fluentd pod logs

Additional info:
*******************************************************************************
Work around is use the same "log-driver" in  /etc/docker/daemon.json or /etc/sysconfig/docker  with `docker info | grep 'Logging Driver'`

Comment 3 Junqi Zhao 2018-03-06 02:38:11 UTC
Issue is not fixed, it is changed to ON_QA by errata, change back to MODIFIED now

Comment 4 Jeff Cantrill 2018-03-06 14:16:43 UTC
This fix is available in image  logging-fluentd-docker-v3.7.26-1 and later: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=647573

Sam, can you help out Junqi?

Comment 5 Junqi Zhao 2018-03-07 01:19:02 UTC
(In reply to Jeff Cantrill from comment #4)
> This fix is available in image  logging-fluentd-docker-v3.7.26-1 and later:
> https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=647573
> 
> Sam, can you help out Junqi?

Sam

This is because issue is related to system container docker and issue is not fixed. see https://bugzilla.redhat.com/show_bug.cgi?id=1551870#c2

Comment 7 Junqi Zhao 2018-03-30 08:39:13 UTC
Tested with logging-fluentd/images/v3.7.42-2, issue is not fixed, use system container docker,there are still continuously info "record cannot use elasticsearch index name type project_full" 

Move to Assigned

Comment 8 Rich Megginson 2018-03-30 15:24:29 UTC
(In reply to Junqi Zhao from comment #7)
> Tested with logging-fluentd/images/v3.7.42-2, issue is not fixed, use system
> container docker,there are still continuously info "record cannot use
> elasticsearch index name type project_full" 
> 
> Move to Assigned

If you do _not_ use system container docker, does it work?  That is, if you use "regular" docker, is the bug fixed?  If so, I would like to move it to the bucket of bugs we have related to system container docker.

Comment 10 Junqi Zhao 2018-04-03 03:06:14 UTC
(In reply to Rich Megginson from comment #8)
> (In reply to Junqi Zhao from comment #7)
> > Tested with logging-fluentd/images/v3.7.42-2, issue is not fixed, use system
> > container docker,there are still continuously info "record cannot use
> > elasticsearch index name type project_full" 
> > 
> > Move to Assigned
> 
> If you do _not_ use system container docker, does it work?  That is, if you
> use "regular" docker, is the bug fixed?  If so, I would like to move it to
> the bucket of bugs we have related to system container docker.

"regular" docker does not have this issue, this issue is only happen with system container docker


Note You need to log in before you can comment on or make changes to this bug.