Setting Up Monitoring Akka Streamlets

Cloudflow allows you to use telemetry just by adding the necessary configuration. Telemetry provides instrumentation to report metrics for Akka components through different plugins, such as Prometheus or Elasticsearch. These metrics can then be accessed through a Grafana dashboard, for instance.

In this case, we’ll describe the use case where the reporter is Prometheus and the dashboard is Grafana. Same as in Cloudflow Console. The required configuration is, as per documentation, explained in following sections.

Sbt-cinnamon plugin

It’s required to add sbt-cinnamon plugin into plugins.sbt file located in the project folder. In order to do this we need to add the following:

addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % [version])

This setting would need to change version to latest

Library dependencies

It’s required to add the libraries in charge of instrumenting each Akka component and the Prometheus backend plugin and HTTP server in the build.sbt. This is done by adding the following:

Cinnamon.library.cinnamonAgent,
Cinnamon.library.cinnamonAkka,
Cinnamon.library.cinnamonAkkaStream,
Cinnamon.library.cinnamonAkkaHttp,
Cinnamon.library.cinnamonPrometheus,
Cinnamon.library.cinnamonPrometheusHttpServer

The libraries cinnamonAkka, cinnamonAkkaStream, and cinnamonAkkaHttp are in charge of adding the metrics, also called instrumentations. While the libraries cinnamonPrometheus and cinnamonPrometheusServer provide the Prometheus backend plugin. Finally, cinnamonAgent adds the cinnamon-agent-[version].jar library that we have to point to when deploying the app. This last point is further explained at the end of this section, in "Deployment configuration".

Cinnamon settings

The following settings are required per project configuration in the build.sbt. This must be added to each project with CloudflowAkkaPlugin. This typically would be set in some common settings and reused in each project.

  cinnamonSuppressRepoWarnings := true,
  cinnamon in test := true,
  cinnamon in run := true,
  cinnamonLogLevel := "INFO",

Deployment configuration

It is required to create a config file (dev.conf, for instance) to be used when deploying as kubectl cloudflow deploy target/[project-name].json --conf dev.conf. The contents of this config file is explained in the following two sections.

The Cinnamon configuration for telemetry.

The following configuration is directly in line with the Cinnamon libraries we added as dependencies in the build.sbt. We are here configuring the streams instrumentation, and we can do so because we previously added in the Cinnamon.library.cinnamonAkkaStream library dependency in build.sbt.

cloudflow.runtimes.akka {
  config {
    cinnamon.prometheus {
      exporters += http-server
    }
    cinnamon.akka {
      streams {
        "*" {
          report-by = instance
        }
      }
    }
  }
}

Opening port and setting -javaagent

As we have previously chosen Cinnamon.library.cinnamonPrometheusHttpServer we now need to allow access the Prometheus HTTP server exporter. It’s also necessary to set cinnamon-agent.jar as a java agent. The following configuration shows both.

cloudflow.runtimes.akka {
  kubernetes.pods.pod {
    containers.container {
      ports = [
        {
          container-port = 9001
          name = "c-metrics"
        }
      ]
      env = [
        { name = "JAVA_OPTS"
          value = "-javaagent:/opt/cloudflow/cinnamon-agent-[version].jar"
        }
      ]
    }
  }
}

The name of the port has to end in metrics. The Prometheus HTTP server exporter port is 9001 by default. Streamlet pods are already annotated with prometheus.io/scrape set to true. This configuration is allowing port 9001 to be scraped by Prometheus. If you’re using your own Prometheus installation you can change this default like (this and then configure your Prometheus to scrape whichever port you have chosen.

The full path to the cinnamon-agent jar file needs to be specified in the -javaagent setting. The sbt build copies the cinnamon java agent dependency to /opt/cloudflow in the image. Setting -javaagent:/opt/cloudflow/cinnamon-agent-[version].jar as JAVA_OPTS allows the cinnamon agent to instrument the streamlets and capture the metrics. The Prometheus HTTP server exporter makes the metrics available to Prometheus.