Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1514580 - [starter-us-east-1] Logs being spammed by RBAC DENY
Summary: [starter-us-east-1] Logs being spammed by RBAC DENY
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Auth
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 4.1.0
Assignee: Mo
QA Contact: Chuan Yu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-17 19:39 UTC by Justin Pierce
Modified: 2019-04-05 13:48 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-05 13:48:12 UTC


Attachments (Terms of Use)

Description Justin Pierce 2017-11-17 19:39:02 UTC
Description of problem:
After upgrading to v3.7.9, master logs are being spammed with a huge number of entries. See attachement

Version-Release number of selected component (if applicable):
v3.7.9

How reproducible:
100%

Comment 2 Simo Sorce 2017-11-17 20:03:04 UTC
Justin,
do you know what is causing them ?

Comment 3 Mo 2017-11-18 18:42:11 UTC
Has something modified the build controller's permissions on this cluster?

It has the following by default (which includes the ability to get SAs across all namespaces):



$ oc adm policy who-can get serviceaccount --all-namespaces 
Namespace: <all>
Verb:      get
Resource:  serviceaccounts

Users:  system:admin
        system:kube-controller-manager
        system:serviceaccount:default:pvinstaller
        system:serviceaccount:kube-system:generic-garbage-collector
        system:serviceaccount:kube-system:namespace-controller
        system:serviceaccount:openshift-infra:build-controller
        system:serviceaccount:openshift-infra:serviceaccount-controller
        system:serviceaccount:openshift-infra:serviceaccount-pull-secrets-controller
        system:serviceaccount:openshift-infra:template-instance-controller

Groups: system:cluster-admins
        system:cluster-readers
        system:masters



$ oc describe clusterrole.rbac system:openshift:controller:build-controller
Name:		system:openshift:controller:build-controller
Labels:		<none>
Annotations:	authorization.openshift.io/system-only=true
		rbac.authorization.kubernetes.io/autoupdate=true
PolicyRule:
  Resources						Non-Resource URLs	Resource Names	Verbs
  ---------						-----------------	--------------	-----
  buildconfigs						[]			[]		[get]
  buildconfigs.build.openshift.io			[]			[]		[get]
  builds						[]			[]		[delete get list patch update watch]
  builds.build.openshift.io				[]			[]		[delete get list patch update watch]
  builds/custom						[]			[]		[create]
  builds.build.openshift.io/custom			[]			[]		[create]
  builds/docker						[]			[]		[create]
  builds.build.openshift.io/docker			[]			[]		[create]
  builds/finalizers					[]			[]		[update]
  builds.build.openshift.io/finalizers			[]			[]		[update]
  builds/jenkinspipeline				[]			[]		[create]
  builds.build.openshift.io/jenkinspipeline		[]			[]		[create]
  builds/optimizeddocker				[]			[]		[create]
  builds.build.openshift.io/optimizeddocker		[]			[]		[create]
  builds/source						[]			[]		[create]
  builds.build.openshift.io/source			[]			[]		[create]
  configmaps						[]			[]		[get list]
  events						[]			[]		[create patch update]
  imagestreams						[]			[]		[get list]
  imagestreams.image.openshift.io			[]			[]		[get list]
  namespaces						[]			[]		[get]
  pods							[]			[]		[create delete get list]
  podsecuritypolicysubjectreviews			[]			[]		[create]
  podsecuritypolicysubjectreviews.security.openshift.io	[]			[]		[create]
  secrets						[]			[]		[get list]
  serviceaccounts					[]			[]		[get list]



$ oc describe clusterrolebinding.rbac system:openshift:controller:build-controller
Name:		system:openshift:controller:build-controller
Labels:		<none>
Annotations:	rbac.authorization.kubernetes.io/autoupdate=true
Role:
  Kind:	ClusterRole
  Name:	system:openshift:controller:build-controller
Subjects:
  Kind			Name			Namespace
  ----			----			---------
  ServiceAccount	build-controller	openshift-infra



What is the output of the above commands on starter-us-east-1?

Comment 6 Simo Sorce 2017-12-15 14:49:36 UTC
Can we get more info here ? If no more details are available I will have to close with insufficient data.

Comment 8 Simo Sorce 2018-01-03 17:30:34 UTC
We have no way to further analyze this issue, closing for now, can be reopenend if it happens again and more data is available.

Comment 9 Clayton Coleman 2018-01-03 17:41:32 UTC
Please indicate whether you tracked down from the Online team whether a config change for permissions was put into place.  Something was obviously broken with RBAC, and it looks like no one understands how it was fixed?

Comment 10 Mo 2018-01-03 19:32:15 UTC
Since this would likely have been caused by someone messing with the bootstrap roles / bindings, a restart of any 3.7 master would have caused it to be fixed by auto reconciliation.

Comment 12 Simo Sorce 2018-01-08 14:25:12 UTC
Reopening,
Mo can you work with Stefanie to find out what's up on that cluster ?


Note You need to log in before you can comment on or make changes to this bug.