Set up Alertmanager

Lightbend Console uses Prometheus Alertmanager for routing alerts that the Console generates. Alertmanager is able to route alerts to many different integration points, including Slack, PagerDuty, and others. It can also route based on severity and workload. Console installs an Alertmanager by default, but you can use your own instead of the default.

The Alertmanager UI is available at /service/alertmanager, relative to the Lightbend Console URL. On a minikube install, this will be at http://192.168.99.100:30080/service/alertmanager.

For more details on Alertmanager and other alert integration, see:

Configuring the default Alertmanager

The default installation of Lightbend Console comes with an Alertmanager deployment. To make it more useful for your project, you will have to do some additional configuration. The Alertmanager bundled in Console is bound to a ConfigMap for its configuration file.

To set up Alertmanager for your own use case, you need to:

  1. Create an Alertmanager configuration file.
  2. Load it into Kubernetes as a ConfigMap.
  3. Pass the ConfigMap name to the Console installer script.

Once done, Alertmanager automatically picks up any updates to the ConfigMap. The following sections describe these steps in detail.

1. Create the configuration file

Create an alertmanager.yml file with your desired configuration. The config file must be named alertmanager.yml, since the Alertmanager in Lightbend Console has been configured to use this name. We provide an example as a starting point. See the official documentation for more details on configuring this file.

Lightbend Console will generate alerts based on the configured monitors. These alerts have labels that can be used for the routing rules. The example illustrates how to use these labels. The available labels are:

Label Key Description
es_workload The workload in Lightbend Console. This can be used to route alerts to different groups based on workload.
severity Either warning or critical. This can be used to route alerts based on their severity.

2. Create the ConfigMap

Next, create a Kubernetes ConfigMap with the configuration file.

Move alertmanager.yml and any required .tmpl files into a new directory:

mkdir alertmanager-config
cp alertmanager.yml alertmanager-config/
cp *.tmpl alertmanager-config/

Then create a ConfigMap in the Lightbend Console namespace. Here we use the lightbend namespace:

kubectl -n lightbend create configmap my-alertmanager-config --from-file=alertmanager-config/ --dry-run -o yaml | kubectl apply -f -

3. Configure Lightbend Console

The last step is to configure Lightbend Console’s Alertmanager to use the new ConfigMap.

Pass the argument --set alertManagerConfigMap=my-alertmanager-config to lbc.py. The installer script is safe to call multiple times, so if you’ve already installed Lightbend Console you can re-run it.

To confirm Alertmanager is using the new configuration, check the Status link on the Alertmanager UI.

Coordinating Alertmanager and Prometheus links

This step is optional.

Alerts generated by Alertmanager (e.g. email alerts) typically contain a link into the Alertmanager UI that describes the alert that fired. The Alertmanager pages also contain (“Source)” links back to the Prometheus data that triggered the alert. These links assume that both the Alertmanager and Prometheus are externally accessible so will be broken for typical installations of the Console.

If you want to make these links work, you need to make the Console externally accessible (e.g. using a proxy) and set the esConsoleURL Console config accordingly. Consider the security implications of making the Console externally accessible before doing so.

If used, the esConsoleURL config should be set to the URL used for external access to the console. For example, if external access to the console via an ingress is available at console.mycorp.com:8080, then pass the argument --set esConsoleURL=http://console.mycorp.com:8080 to lbc.py.

Updating Alertmanager configuration

Once you’ve configured the Console to use your ConfigMap, any future changes will be automatically picked up by Alertmanager. There may be a small delay before Alertmanager reloads, because we rely on Kubernetes to periodically synchronize the ConfigMap contents. This can take up to a minute.

You can repeat the steps described in 2.Create the ConfigMap to update the ConfigMap as many times as you’d like. Another approach is to edit the ConfigMap directly with kubectl:

    kubectl edit -n lightbend configmap my-alertmanager-config

Using existing Alertmanager(s)

If you already have an Alertmanager installed somewhere, you can use it instead of Console’s default Alertmanager. It allows for running Alertmanagers in different namespaces or outside of Kubernetes. In addition to that, you can use High-Availability Alertmanager cluster to improve reliability.

Installing with existing Alertmanager

To use an existing Alertmanager, you need to pass in values to the Console lbc.py script. Specifically, set the createAlertManger Helm chart value to false and provide an address with a port number where your Alertmanager can be reached:

./lbc.py install --namespace=lightbend --version=$es.version$ --set createAlertManager=false --set alertManagers=my-alertmanager:9093

In the above example, my-alertmanager is the Kubernetes service name given to an Alertmanager already running in the lightbend namespace. For connecting to an Alertmanager in a different namespace, use a full Kubernetes service DNS name like my-alertmanager.monitoring.svc.cluster.local:9093. More about this can be found in Kubernetes docs.

Console connects to your specified Alertmanager through HTTP by doing a DNS lookup for a given name. That’s why you can use Kubernetes service names - they get registered in the internal cluster DNS registry. It also means you can use an Alertmanager running outside of the cluster where Console is installed. It’s even possible to specify an IPv4 or IPv6 address:

./lbc.py install --namespace=lightbend --version=$es.version$ --set createAlertManager=false --set alertManagers=10.2.0.64:9093

Installing with Alertmanager cluster

Prometheus documentation describes how to set up a high-availability cluster of Alertmanagers. Internally, Alertmanager cluster takes care of deduplicating, silencing, and triggering configured behaviours just as if running a single instance. To use such a cluster, you need to provide addresses for all its instances to the Console install script:

./lbc.py install --namespace=lightbend --version=$es.version$ --set createAlertManager=false --set alertManagers=alertmanager-00:9093,alertmanager-01:9093,alertmanager-02:9093

Configuring existing Alertmanager(s)

Note

It’s possible to run your own Alertmanager in many different configurations, so consult with Prometheus docs on the best ways to set it up.

In the case your Alertmanager is running in the same Kubernetes cluster, you can use ConfigMap for configuring it, as described in Configuration overview. This requires modifying the Deployment resource to reference the right ConfigMap and reload it when changes occur. While a detailed description is out of scope for this document, you can look at the included Alertmanager Deployment template as an example.