Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1363932 - [intservice_public_138] Failed to show logs in kibana console with log driver used journald when OSE is containerized installation
Summary: [intservice_public_138] Failed to show logs in kibana console with log driver...
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Luke Meyer
QA Contact: chunchen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-04 03:35 UTC by chunchen
Modified: 2016-09-30 02:15 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-08 21:00:09 UTC


Attachments (Terms of Use)
kibana console screenshot (deleted)
2016-08-04 03:35 UTC, chunchen
no flags Details
deployer logs (deleted)
2016-08-05 06:14 UTC, chunchen
no flags Details
fluentd log (deleted)
2016-08-05 06:14 UTC, chunchen
no flags Details

Description chunchen 2016-08-04 03:35:02 UTC
Description of problem:
It's failed to show logs in kibana console with log driver used journald when OSE is containerized installation,but it's not reproduced with rpm installation.

Version-Release number of selected component (if applicable):
brew-pulp-docker01.web.prod..com:8888/openshift3/logging-elasticsearch   3.3.0               e71d2b04669c
brew-pulp-docker01.web.prod..com:8888/openshift3/logging-fluentd         3.3.0               80847240fa91
brew-pulp-docker01.web.prod..com:8888/openshift3/logging-deployer        3.3.0               1c127f4f36a0
brew-pulp-docker01.web.prod..com:8888/openshift3/logging-curator         3.3.0               2c88e1273c11
brew-pulp-docker01.web.prod..com:8888/openshift3/logging-auth-proxy      3.3.0               c0b7d9b08a2e
brew-pulp-docker01.web.prod..com:8888/openshift3/logging-kibana          3.3.0               32d276bb46ae

openshift v3.3.0.13
kubernetes v1.3.0+57fb9ac
etcd 2.3.0+git

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.10.x86_64
 Go version:      go1.6.2
 Git commit:      2a93377-unsupported
 Built:           Fri Jul 29 13:45:25 2016
 OS/Arch:         linux/amd64


How reproducible:
Always

Steps to Reproduce:
1. Install OSE env via containerized method

2. Make docker daemon to use journald as log driver on node machine
# ps -ef |grep docker

3. Deploy logging stack with setting "use-journal=true" according to https://github.com/openshift/origin-aggregated-logging/tree/master/deployer#using-the-logging-deployer

Below is the creating configmap command line:
$ oc create configmap logging-deployer  --from-literal kibana-hostname=kibana.0803-mh8.qe.rhcloud.com --from-literal public-master-url=https://openshift-126...com:8443 --from-literal es-cluster-size=1 --from-literal enable-ops-cluster=false --from-literal use-journal=true

Below is the creating pod command line:
$ oc new-app logging-deployer-template -p IMAGE_PREFIX=brew-pulp-docker01.web.prod...com:8888/openshift3/,IMAGE_VERSION=3.3.0,MODE=install

4. Check the pods and configmap
$ oc get pods -o wide
$ oc get configmap logging-deployer -o yaml

5. Check the rubygem/rpm packages in fluentd container
$ oc rsh logging-fluentd-4xlg3

6. Try to check the pod logs in kibana console via browser

Actual results:
at step 2:
root     11903     1  8 21:49 ?        00:01:34 /usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --selinux-enabled --insecure-registry=172.30.0.0/16 --log-driver=journald --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/rhel-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true -b=lbr0 --mtu=1450 --add-registry brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 --add-registry registry.access.redhat.com --insecure-registry virt-openshift-05.lab.eng.nay.redhat.com:5000 --insecure-registry virt-openshift-05.lab.eng.nay.redhat.com:5001 --insecure-registry brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 --insecure-registry registry.qe.openshift.com

root     12068     1  0 21:49 ?        00:00:00 /usr/bin/docker-current run --name atomic-openshift-node --rm --privileged --net=host --pid=host --env-file=/etc/sysconfig/atomic-openshift-node -v /:/rootfs:ro -e CONFIG_FILE=/etc/origin/node/node-config.yaml -e OPTIONS=--loglevel=5 -e HOST=/rootfs -e HOST_ETC=/host-etc -v /var/lib/origin:/var/lib/origin:rslave -v /etc/origin/node:/etc/origin/node -v /etc/localtime:/etc/localtime:ro -v /etc/machine-id:/etc/machine-id:ro -v /run:/run -v /sys:/sys:ro -v /usr/bin/docker:/usr/bin/docker:ro -v /var/lib/docker:/var/lib/docker -v /lib/modules:/lib/modules -v /etc/origin/openvswitch:/etc/openvswitch -v /etc/origin/sdn:/etc/openshift-sdn -v /etc/systemd/system:/host-etc/systemd/system -v /var/log:/var/log -v /dev:/dev --volume=/usr/bin/docker-current:/usr/bin/docker-current:ro --volume=/etc/sysconfig/docker:/etc/sysconfig/docker:ro openshift3/node:v3.3.0.13

at step 4:
$ oc get pod -o wide
NAME                          READY     STATUS      RESTARTS   AGE       IP          NODE
logging-curator-1-f371u       1/1       Running     0          5m        10.1.1.4    openshift-147.lab.sjc.redhat.com
logging-deployer-4ck4z        0/1       Completed   0          7m        10.1.1.4    openshift-147.lab.sjc.redhat.com
logging-es-foe2d5ab-1-phly1   1/1       Running     0          5m        10.1.1.9    openshift-147.lab.sjc.redhat.com
logging-fluentd-4xlg3         1/1       Running     0          5m        10.1.1.5    openshift-147.lab.sjc.redhat.com
logging-kibana-1-w4owb        2/2       Running     0          5m        10.1.1.10   openshift-147.lab.sjc.redhat.com

$ oc get configmap logging-deployer -o yaml
apiVersion: v1
data:
  enable-ops-cluster: "false"
  es-cluster-size: "1"
  kibana-hostname: kibana.0803-mh8.qe.rhcloud.com
  public-master-url: https://openshift-126.lab.sjc.redhat.com:8443
  use-journal: "true"
kind: ConfigMap
metadata:
  creationTimestamp: 2016-08-04T01:57:06Z
  name: logging-deployer
  namespace: clogg
  resourceVersion: "17772"
  selfLink: /api/v1/namespaces/clogg/configmaps/logging-deployer
  uid: bf0661b4-59e6-11e6-a394-fa163ee090d2

at step 5:
$ oc rsh logging-fluentd-4xlg3
sh-4.2# gem list |grep -e jour -e fluen
rpm -fluent-plugin-docker_metadata_filter (0.1.1)
fluent-plugin-elasticsearch (1.3.0)
fluent-plugin-flatten-hash (0.2.0)
fluent-plugin-kubernetes_metadata_filter (0.24.0)
fluent-plugin-rewrite-tag-filter (1.5.5)
fluent-plugin-systemd (0.0.3)
fluentd (0.12.20)
systemd-journal (1.2.2)

sh-4.2# rpm -qa | grep -e jour -e fluent
rubygem-systemd-journal-1.2.2-3.el7.noarch
fluentd-0.12.20-1.el7.noarch
rubygem-fluent-plugin-elasticsearch-1.3.0-2.el7.noarch
rubygem-fluent-plugin-flatten-hash-0.2.0-1.el7aos.noarch
rubygem-fluent-plugin-docker_metadata_filter-0.1.1-1.el7aos.noarch
rubygem-fluent-plugin-kubernetes_metadata_filter-0.24.0-1.el7.noarch
rubygem-fluent-plugin-systemd-0.0.3-2.el7.noarch
rubygem-fluent-plugin-rewrite-tag-filter-1.5.5-2.el7.noarch

at step 6: No logs shown in the kibana console, please refer to the screenshot

Expected results:
Should show logs in kibana console with log driver used journald when OSE is containerized installation

Additional info:
On node machine:
root@openshift-147 ~]# ls /var/log/journal* -ltr
-rw-r--r--. 1 root root 124 Aug  3 22:35 /var/log/journal.pos

[root@openshift-147 ~]# ls /run/log/journal* -ltr
total 0
drwxr-s---+ 2 root systemd-journal 200 Aug  3 22:30 d9ae96c9a4a24a52ad714d489bb55293

[root@openshift-147 ~]# ls /run/log/journal//d9ae96c9a4a24a52ad714d489bb55293/*
/run/log/journal//d9ae96c9a4a24a52ad714d489bb55293/system@f9f266edd25f4c99a759b83abb1fef61-000000000007d545-00053932b76358a8.journal
/run/log/journal//d9ae96c9a4a24a52ad714d489bb55293/system@f9f266edd25f4c99a759b83abb1fef61-000000000008445a-000539334b612031.journal
/run/log/journal//d9ae96c9a4a24a52ad714d489bb55293/system@f9f266edd25f4c99a759b83abb1fef61-000000000008b36f-00053933df77b7f2.journal
/run/log/journal//d9ae96c9a4a24a52ad714d489bb55293/system@f9f266edd25f4c99a759b83abb1fef61-000000000009227f-0005393472fcc6aa.journal
/run/log/journal//d9ae96c9a4a24a52ad714d489bb55293/system@f9f266edd25f4c99a759b83abb1fef61-0000000000099194-0005393506d96ba8.journal
/run/log/journal//d9ae96c9a4a24a52ad714d489bb55293/system@f9f266edd25f4c99a759b83abb1fef61-00000000000a00a4-000539354f8f920b.journal
/run/log/journal//d9ae96c9a4a24a52ad714d489bb55293/system@f9f266edd25f4c99a759b83abb1fef61-00000000000a6fb4-00053935847a3e2e.journal
/run/log/journal//d9ae96c9a4a24a52ad714d489bb55293/system.journal

Comment 1 chunchen 2016-08-04 03:35:43 UTC
Created attachment 1187333 [details]
kibana console screenshot

Comment 2 Rich Megginson 2016-08-05 00:42:12 UTC
# show the deployer logs
oc logs logging-deployer-4ck4z

# show the fluentd logs
oc logs logging-fluentd-4xlg3

# see if the files are in the right places
oc exec logging-fluentd-4xlg3 -- ls -alrtF /etc/fluent /etc/fluent/configs.d /etc/fluent/configs.d/dynamic

# see if configured to use journal
oc exec logging-fluentd-4xlg3 -- cat /etc/fluent/configs.d/dynamic/input-syslog-default-syslog.conf

# see if anything is in elasticsearch
oc exec logging-kibana-1-w4owb -- curl -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key https://logging-es:9200/_search | python -mjson.tool

Comment 3 chunchen 2016-08-05 06:13:07 UTC
For pod logs, please refer to the attachments, below messages are the output of command line:

[chunchen@F17-CCY daily]$ oc exec logging-fluentd-h9qwd -- ls -alrtF /etc/fluent /etc/fluent/configs.d /etc/fluent/configs.d/dynamic
/etc/fluent/configs.d:
total 8
drwxr-xr-x. 2 root root 4096 Jul 26 15:49 openshift/
drwxr-xr-x. 5 root root   47 Jul 26 15:51 ./
drwxrwxrwx. 3 root root 4096 Aug  5 01:21 user/
drwxr-xr-x. 4 root root   51 Aug  5 01:21 ../
drwxrwxrwx. 2 root root  101 Aug  5 01:21 dynamic/

/etc/fluent:
total 4
lrwxrwxrwx.  1 root root   38 Jul 26 15:51 fluent.conf -> /etc/fluent/configs.d/user/fluent.conf
drwxr-xr-x.  5 root root   47 Jul 26 15:51 configs.d/
drwxrwxrwt.  3 root root  140 Aug  5 01:21 keys/
drwxr-xr-x. 51 root root 4096 Aug  5 01:21 ../
drwxr-xr-x.  4 root root   51 Aug  5 01:21 ./

/etc/fluent/configs.d/dynamic:
total 12
drwxr-xr-x. 5 root root  47 Jul 26 15:51 ../
-rw-r--r--. 1 root root 150 Aug  5 01:21 input-syslog-default-syslog.conf
-rw-r--r--. 1 root root   1 Aug  5 01:21 es-copy-config.conf
-rw-r--r--. 1 root root   1 Aug  5 01:21 es-ops-copy-config.conf
drwxrwxrwx. 2 root root 101 Aug  5 01:21 ./



[chunchen@F17-CCY daily]$ oc exec logging-fluentd-h9qwd -- cat /etc/fluent/configs.d/dynamic/input-syslog-default-syslog.conf
<source>
  @type systemd
  @label @INGRESS
  path "/run/log/journal"
  pos_file /var/log/journal.pos
  tag journal
  read_from_head "false"
</source>



[chunchen@F17-CCY daily]$ oc exec logging-kibana-1-v2lgw  -- curl -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key https://logging-es:9200/_search | python -mjson.tool
{
    "error": "ForbiddenException[Attempt from null to _all indices for indices:data/read/search and User [name=system.logging.kibana, roles=[]]]", 
    "status": 403
}

Comment 4 chunchen 2016-08-05 06:14:04 UTC
Created attachment 1187750 [details]
deployer logs

Comment 5 chunchen 2016-08-05 06:14:35 UTC
Created attachment 1187751 [details]
fluentd log

Comment 6 chunchen 2016-08-05 06:27:57 UTC
After I re-installed a fresh OSE env and used latest images, the issue did not reproduce any more.

Could you help to change the bug's status to ON_QA, then I can mark it as verified?

Thanks!

Comment 7 chunchen 2016-08-08 02:05:41 UTC
According to comment #6, mark it as verified.


Note You need to log in before you can comment on or make changes to this bug.