Console RBAC usage

Lightbend Console functionality relies on gathering metrics inside the Kubernetes cluster. That means a certain level of access is needed for Console to work. This section describes in detail what specifically is accessed and how.

High-level overview

The main components of Lightbend Console are Prometheus, Grafana, and Console UI itself. Of these, Prometheus uses further subcomponents: Alertmanager, Kubernetes State Metrics Agent. How they all fit together is explained in the architecture overview. Here instead we’ll focus on privileges these components have inside the Kubernetes cluster.

Grafana, Console UI and Alertmanager need no special privileges because they only consume data provided to them by Prometheus. Console UI, Alertmanager, and Grafana have their own PersistentVolumes for storing configuration.

All the monitoring data is actually gathered by Prometheus. It acts under a Kubernetes ServiceAccount named prometheus-server that is bound to a ClusterRole with read-only permissions for discovering workloads across the cluster. Furthermore, Prometheus uses data from Kubernetes State Metrics Agent. Kubernetes State Metrics has its own ClusterRole that permits read-only access to Kubernetes cluster state.

Prometheus Server

Prometheus can automatically discover relevant Kubernetes resources: nodes, services, pods, endpoints and ingresses. More in-depth description of this can be found in the official docs. Lightbend Console has a ClusterRole for Prometheus which allows it to discover said resources. On top of that, Prometheus gathers metrics from workloads by scraping their /metrics URLs, so it also has permission to do GET requests to /metrics even when no Kubernetes Services are set up to expose it as a resource. The corresponding rules for Prometheus ClusterRole resource are these:

rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

Kubernetes State Metrics

Because Prometheus itself has no deeper understanding of Kubernetes concepts besides being able to discover its resources as services for scraping, Lightbend Console installs Kubernetes State Metrics component for tracking cluster-wide metrics. For this to work, it needs to see all resources in the cluster. It runs under service account named prometheus-kube-state-metrics. The rule description for the ClusterRole looks like this:

rules:
- apiGroups: [""]
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs: ["list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]