Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1513408

Summary: Continuously error info "record cannot use elasticsearch index name type project_full" in fluentd pod logs if use system container docker.
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED WONTFIX QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: low    
Version: 3.7.0CC: aos-bugs, gferrazs, jcantril, pweil, rmeggins, smunilla, veli-matti.leppanen
Target Milestone: ---Flags: jcantril: needinfo? (smunilla)
gferrazs: needinfo? (jcantril)
Target Release: 3.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Messages for which the unique namespace id could not be determined could not be properly indexed. Consequence: Messages could be lost and the error message appears in the logs Fix: Modify the cache algorithm to provide the necessary data or default the value to 'orphaned'. Result: The error message is resolved and messages are stored in an 'orphaned' index when a unique namespace id can not be determined.
Story Points: ---
Clone Of:
: 1551870 (view as bug list) Environment:
Last Closed: 2019-03-06 20:39:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1551870    

Description Junqi Zhao 2017-11-15 11:13:31 UTC
Description of problem:
Use system container docker,there are continuously info "record cannot use elasticsearch index name type project_full" in fluentd pod logs. 

This issue is only happen with system container docker, since its configure file is /etc/docker/container-daemon.json rather than  /etc/docker/daemon.json or /etc/sysconfig/docker.

Trace:
2017-11-15 10:19:25 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {"message"=>"2017-11-15 10:19:24 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {\"message\"=>\"2017-11-15T10:06:38.903902551Z AUDIT: id=\\\"968849aa-d7a7-45f2-9243-5ece6804e21f\\\" ip=\\\"10.130.0.1\\\" method=\\\"GET\\\" user=\\\"system:anonymous\\\" groups=\\\"\\\\\\\"system:unauthenticated\\\\\\\"\\\" as=\\\"<self>\\\" asgroups=\\\"<lookup>\\\" namespace=\\\"<none>\\\" uri=\\\"/healthz\\\"\\n\", \"level\"=>\"info\", \"hostname\"=>\"localhost.localdomain\", \"pipeline_metadata\"=>{\"collector\"=>{\"ipaddr4\"=>\"10.130.0.16\", \"ipaddr6\"=>\"fe80::c432:91ff:fe2f:d303\", \"inputname\"=>\"fluent-plugin-systemd\", \"name\"=>\"fluentd\", \"received_at\"=>\"2017-11-15T10:19:24.305099+00:00\", \"version\"=>\"0.12.39 1.6.0\"}}, \"@timestamp\"=>\"2017-11-15T10:06:38.904126+00:00\"}\n", "level"=>"info", "hostname"=>"localhost.localdomain", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.130.0.16", "ipaddr6"=>"fe80::c432:91ff:fe2f:d303", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2017-11-15T10:19:25.492302+00:00", "version"=>"0.12.39 1.6.0"}}, "@timestamp"=>"2017-11-15T10:19:24.308025+00:00"}
2017-11-15 10:19:25 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {"message"=>"2017-11-15 10:19:24 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {\"message\"=>\"2017-11-15T10:06:48.906137559Z AUDIT: id=\\\"73f3c8fa-f5a2-4160-b62f-747398429f3a\\\" ip=\\\"10.130.0.1\\\" method=\\\"GET\\\" user=\\\"system:anonymous\\\" groups=\\\"\\\\\\\"system:unauthenticated\\\\\\\"\\\" as=\\\"<self>\\\" asgroups=\\\"<lookup>\\\" namespace=\\\"<none>\\\" uri=\\\"/healthz\\\"\\n\", \"level\"=>\"info\", \"hostname\"=>\"localhost.localdomain\", \"pipeline_metadata\"=>{\"collector\"=>{\"ipaddr4\"=>\"10.130.0.16\", \"ipaddr6\"=>\"fe80::c432:91ff:fe2f:d303\", \"inputname\"=>\"fluent-plugin-systemd\", \"name\"=>\"fluentd\", \"received_at\"=>\"2017-11-15T10:19:24.305285+00:00\", \"version\"=>\"0.12.39 1.6.0\"}}, \"@timestamp\"=>\"2017-11-15T10:06:48.906337+00:00\"}\n", "level"=>"info", "hostname"=>"localhost.localdomain", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.130.0.16", "ipaddr6"=>"fe80::c432:91ff:fe2f:d303", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2017-11-15T10:19:25.492472+00:00", "version"=>"0.12.39 1.6.0"}}, "@timestamp"=>"2017-11-15T10:19:24.308032+00:00"}

******************************************************************************


# docker info | grep 'Logging Driver'
Logging Driver: json-file

# cat /etc/docker/daemon.json 
{}

#cat /etc/sysconfig/docker
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'



Version-Release number of selected component (if applicable):
logging componet version is v3.7.8-1
# openshift version
openshift v3.7.8
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8


How reproducible:
Always

Steps to Reproduce:
1. Deploy logging 3.7


Actual results:
continuously info "record cannot use elasticsearch index name type project_full" in fluentd pod logs

Expected results:
Should not be error in fluentd pod logs

Additional info:
*******************************************************************************
Work around is use the same "log-driver" in  /etc/docker/daemon.json or /etc/sysconfig/docker  with `docker info | grep 'Logging Driver'`

Comment 3 Junqi Zhao 2018-03-06 02:38:11 UTC
Issue is not fixed, it is changed to ON_QA by errata, change back to MODIFIED now

Comment 4 Jeff Cantrill 2018-03-06 14:16:43 UTC
This fix is available in image  logging-fluentd-docker-v3.7.26-1 and later: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=647573

Sam, can you help out Junqi?

Comment 5 Junqi Zhao 2018-03-07 01:19:02 UTC
(In reply to Jeff Cantrill from comment #4)
> This fix is available in image  logging-fluentd-docker-v3.7.26-1 and later:
> https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=647573
> 
> Sam, can you help out Junqi?

Sam

This is because issue is related to system container docker and issue is not fixed. see https://bugzilla.redhat.com/show_bug.cgi?id=1551870#c2

Comment 7 Junqi Zhao 2018-03-30 08:39:13 UTC
Tested with logging-fluentd/images/v3.7.42-2, issue is not fixed, use system container docker,there are still continuously info "record cannot use elasticsearch index name type project_full" 

Move to Assigned

Comment 8 Rich Megginson 2018-03-30 15:24:29 UTC
(In reply to Junqi Zhao from comment #7)
> Tested with logging-fluentd/images/v3.7.42-2, issue is not fixed, use system
> container docker,there are still continuously info "record cannot use
> elasticsearch index name type project_full" 
> 
> Move to Assigned

If you do _not_ use system container docker, does it work?  That is, if you use "regular" docker, is the bug fixed?  If so, I would like to move it to the bucket of bugs we have related to system container docker.

Comment 10 Junqi Zhao 2018-04-03 03:06:14 UTC
(In reply to Rich Megginson from comment #8)
> (In reply to Junqi Zhao from comment #7)
> > Tested with logging-fluentd/images/v3.7.42-2, issue is not fixed, use system
> > container docker,there are still continuously info "record cannot use
> > elasticsearch index name type project_full" 
> > 
> > Move to Assigned
> 
> If you do _not_ use system container docker, does it work?  That is, if you
> use "regular" docker, is the bug fixed?  If so, I would like to move it to
> the bucket of bugs we have related to system container docker.

"regular" docker does not have this issue, this issue is only happen with system container docker