Styjun

I am trying to get heapster eventer to work on a cluster with RBAC enabled. Using the same roles that work for /heapster command does not seem to be sufficient.

On running the pod logs fill up with entries like this:

Failed to load events: events is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list events at the cluster scope

Does anyone know the proper authorization for my heapster service account, short of admin rights?

Eventer deployment doc:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
 labels:
 k8s-app: eventer
 name: eventer
 namespace: kube-system
spec:
 replicas: 1
 selector:
 matchLabels:
 k8s-app: eventer
 strategy:
 rollingUpdate:
 maxSurge: 1
 maxUnavailable: 1
 type: RollingUpdate
 template:
 metadata:
 labels:
 k8s-app: eventer
 spec:
 serviceAccountName: heapster
 containers:
 - name: eventer
 image: k8s.gcr.io/heapster-amd64:v1.5.4
 imagePullPolicy: IfNotPresent
 command:
 - /eventer
 - --source=kubernetes:https://kubernetes.default
 - --sink=log
 resources:
 limits:
 cpu: 100m
 memory: 200Mi
 requests:
 cpu: 100m
 memory: 200Mi
 terminationMessagePath: /dev/termination-log
 restartPolicy: Always
 terminationGracePeriodSeconds: 30

RBAC:

# Original: https://brookbach.com/2018/10/29/Heapster-on-Kubernetes-1.11.3.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: heapster
rules:
 - apiGroups:
 - ""
 resources:
 - pods
 - nodes
 - namespaces
 - events
 verbs:
 - get
 - list
 - watch
 - apiGroups:
 - extensions
 resources:
 - deployments
 verbs:
 - get
 - list
 - update
 - watch
 - apiGroups:
 - ""
 resources:
 - nodes/stats
 verbs:
 - get

Cluster role binding:

# Original: https://github.com/kubernetes-retired/heapster/blob/master/deploy/kube-config/rbac/heapster-rbac.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
 name: heapster
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: heapster
subjects:
 - kind: ServiceAccount
 name: heapster
 namespace: kube-system

Related question:
How to propagate kubernetes events from a GKE cluster to google cloud log

All of the above objects seem to be correct to me.

It's just a hunch but perhaps you created the Deployment first and then the ClusterRole and/or ClusterBindingRole and/or the ServiceAccount itself. Make sure you have these 3 first, then delete the current heapster Pods (or the Deployment, and wait for the Pod to terminate before recreating the Deployment).

(Create the ServiceAccount by kubectl create sa heapster -n kube-system)

Also, you can test if ServiceAccount can list the events by:

kubectl get ev --all-namespaces --as system:serviceaccount:kube-system:heapster