Telemetry OpenTracing

Monitoring Tip of the Month - May 2017

By Hugh McKee, May 19, 2017


We would like to welcome you to the first in a series of monthly articles called The Monitoring Tip of the Month that focuses on the required changes that are necessary to install and configure tracing in an existing Akka project.

In this month’s edition, we examine the following topic: May’s Monitoring Tip of the Month: Telemetry OpenTracing. In future Monitoring Tips of the Month, we will dive deeper into how to use tracing to track activity in Akka systems.


In April 2017 Lightbend Telemetry version 2.4 was released. One of the most significant changes in this release is the addition of OpenTracing integration with support for Jaeger and Zipkin. OpenTracing is a “vendor-neutral open standard for distributed tracing.” Being that Akka is, by its nature is a, platform for building distributed systems tracing is a much needed and often requested feature that is a valuable tool for Akka developers.

This Monitoring Tip of the Month (MTM) focuses on the required changes that are necessary to install and configure tracing in an existing Akka project. In future MTM we will dive deeper into how to use tracing to track activity in Akka systems.

Project Setup

We are going to take an existing Akka project and make the necessary changes to enable Lightbend Telemetry with actor tracing.

We will be following the steps defined in the Lightbend Telemetry setup sbt documentation. The reason for this is that the sample project that we are using is built with SBT. Steps for setting up Maven and Gradle projects are also available.

The project used here can be found in the Lightbend Tech Hub / Get started with Lightbend technologies. The project is “Cluster Java.” Download this project if you wish to follow along. Another option is to take one of your sbt projects and follow along.

As instructed, edit the project/plugins.sbt file.

addSbtPlugin("com.typesafe.sbt" % "sbt-multi-jvm" % "0.3.8")
addSbtPlugin("com.dwijnand" % "sbt-dynver" % "1.1.1")

Add the lines as instructed in the documentation.

addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % "2.4.0")
credentials += Credentials(Path.userHome / ".lightbend" / "commercial.credentials")
resolvers += Resolver.url("lightbend-commercial",
  url("https://repo.lightbend.com/commercial-releases"))(Resolver.ivyStylePatterns)

Here is the complete plugins.sbt file with the initial first 2 lines followed by the added lines.

addSbtPlugin("com.typesafe.sbt" % "sbt-multi-jvm" % "0.3.8")
addSbtPlugin("com.dwijnand" % "sbt-dynver" % "1.1.1")

addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % "2.4.0")
credentials += Credentials(Path.userHome / ".lightbend" / "commercial.credentials")
resolvers += Resolver.url("lightbend-commercial",
  url("https://repo.lightbend.com/commercial-releases"))(Resolver.ivyStylePatterns)

Next make the necessary changes to the build.sbt file. The file initially looked like this.

import com.typesafe.sbt.SbtMultiJvm.multiJvmSettings
import com.typesafe.sbt.SbtMultiJvm.MultiJvmKeys.MultiJvm

val akkaVersion = "2.5.0"

val `akka-sample-cluster-java` = project
  .in(file("."))
  .settings(multiJvmSettings: _*)
  .settings(
    organization := "com.typesafe.akka.samples",
    scalaVersion := "2.12.1",
    scalacOptions in Compile ++= Seq("-deprecation", "-feature", "-unchecked", "-Xlog-reflective-calls", "-Xlint"),
    javacOptions in Compile ++= Seq("-Xlint:unchecked", "-Xlint:deprecation"),
    javacOptions in doc in Compile := Seq("-Xdoclint:none"),
    javaOptions in run ++= Seq("-Xms128m", "-Xmx1024m", "-Djava.library.path=./target/native"),
    libraryDependencies ++= Seq(
      "com.typesafe.akka" %% "akka-actor" % akkaVersion,
      "com.typesafe.akka" %% "akka-remote" % akkaVersion,
      "com.typesafe.akka" %% "akka-cluster" % akkaVersion,
      "com.typesafe.akka" %% "akka-cluster-metrics" % akkaVersion,
      "com.typesafe.akka" %% "akka-cluster-tools" % akkaVersion,
      "com.typesafe.akka" %% "akka-multi-node-testkit" % akkaVersion,
      "org.scalatest" %% "scalatest" % "3.0.1" % Test,
      "io.kamon" % "sigar-loader" % "1.6.6-rev002"),
    fork in run := true,
    mainClass in (Compile, run) := Some("sample.cluster.simple.SimpleClusterApp"),
    // disable parallel tests
    parallelExecution in Test := false,
    licenses := Seq(("CC0", url("http://creativecommons.org/publicdomain/zero/1.0")))
  )
  .configs (MultiJvm)

After adding the monitoring settings the file looks like this. Note that the new and modified lines all end with the “// Telemetry (Cinnamon)” comment.

import com.typesafe.sbt.SbtMultiJvm.multiJvmSettings
import com.typesafe.sbt.SbtMultiJvm.MultiJvmKeys.MultiJvm

val akkaVersion = "2.5.0"

val `akka-sample-cluster-java` = project
  .in(file("."))
  .enablePlugins(Cinnamon) // Telemetry (Cinnamon)
  .settings(multiJvmSettings: _*)
  .settings(
    cinnamon in run := true, // Telemetry (Cinnamon)
    cinnamon in test := true, // Telemetry (Cinnamon)
    cinnamonLogLevel := "INFO", // Telemetry (Cinnamon)

    organization := "com.typesafe.akka.samples",
    scalaVersion := "2.12.1",
    scalacOptions in Compile ++= Seq("-deprecation", "-feature", "-unchecked", "-Xlog-reflective-calls", "-Xlint"),
    javacOptions in Compile ++= Seq("-Xlint:unchecked", "-Xlint:deprecation"),
    javacOptions in doc in Compile := Seq("-Xdoclint:none"),
    javaOptions in run ++= Seq("-Xms128m", "-Xmx1024m", "-Djava.library.path=./target/native"),

    libraryDependencies += Cinnamon.library.cinnamonCHMetrics, // Telemetry (Cinnamon)
    libraryDependencies += Cinnamon.library.cinnamonAkka, // Telemetry (Cinnamon)
    libraryDependencies += Cinnamon.library.cinnamonOpenTracingJaeger, // Telemetry (Cinnamon)
    libraryDependencies += Cinnamon.library.cinnamonOpenTracingZipkin, // Telemetry (Cinnamon)

    libraryDependencies ++= Seq(
      "com.typesafe.akka" %% "akka-actor" % akkaVersion,
      "com.typesafe.akka" %% "akka-remote" % akkaVersion,
      "com.typesafe.akka" %% "akka-cluster" % akkaVersion,
      "com.typesafe.akka" %% "akka-cluster-metrics" % akkaVersion,
      "com.typesafe.akka" %% "akka-cluster-tools" % akkaVersion,
      "com.typesafe.akka" %% "akka-multi-node-testkit" % akkaVersion,
      "org.scalatest" %% "scalatest" % "3.0.1" % Test,
      //      "io.kamon" % "sigar-loader" % "1.6.6-rev002"), // Telemetry (Cinnamon)
      "org.slf4j" % "slf4j-api"       % "1.7.25", // Telemetry (Cinnamon)
      "org.slf4j" % "jcl-over-slf4j"  % "1.7.25"), // Telemetry (Cinnamon)
    fork in run := true,
    mainClass in (Compile, run) := Some("sample.cluster.simple.SimpleClusterApp"),
    // disable parallel tests
    parallelExecution in Test := false,
    licenses := Seq(("CC0", url("http://creativecommons.org/publicdomain/zero/1.0")))
  )
  .configs (MultiJvm)

The changes made to this build.sbt are somewhat different than what is shown in the documentation, but they are equivalent.

Now that the sbt changes have been completed the next thing that needs to be configured is the Akka configuration settings. The initial application.conf before making any monitoring changes is as follows.

akka {
  actor {
    provider = "cluster"
  }
  remote {
    log-remote-lifecycle-events = off
    netty.tcp {
      hostname = "127.0.0.1"
      port = 0
    }
  }

  cluster {
    seed-nodes = [
      "akka.tcp://ClusterSystem@127.0.0.1:2551",
      "akka.tcp://ClusterSystem@127.0.0.1:2552"]

    # auto downing is NOT safe for production deployments.
    # you may want to use it during development, read more about it in the docs.
    auto-down-unreachable-after = 10s
  }
}

# Disable legacy metrics in akka-cluster.
Akka.cluster.metrics.enabled = off

# Enable metrics extension in akka-cluster-metrics.
Akka.extensions = ["akka.cluster.metrics.ClusterMetricsExtension"]

# Sigar native library extract location during tests.
# Note: use per-jvm-instance folder when running multiple jvm on one host. 
Akka.cluster.metrics.native-library-extract-folder = ${user.dir}/target/native<

Let’s make some minimal changes to get actor tracing working. First, there are some required setting for Actor configuration. The following is the complete application.conf file including the monitoring configuration settings added to the end of the file.

akka {
  actor {
    provider = "cluster"
  }
  remote {
    log-remote-lifecycle-events = off
    netty.tcp {
      hostname = "127.0.0.1"
      port = 0
    }
  }

  cluster {
    seed-nodes = [
      "akka.tcp://ClusterSystem@127.0.0.1:2551",
      "akka.tcp://ClusterSystem@127.0.0.1:2552"]

    # auto downing is NOT safe for production deployments.
    # you may want to use it during development, read more about it in the docs.
    auto-down-unreachable-after = 10s
  }
}

# Disable legacy metrics in akka-cluster.
akka.cluster.metrics.enabled=off

# Enable metrics extension in akka-cluster-metrics.
akka.extensions=["akka.cluster.metrics.ClusterMetricsExtension"]

# Sigar native library extract location during tests.
# Note: use per-jvm-instance folder when running multiple jvm on one host. 
akka.cluster.metrics.native-library-extract-folder=${user.dir}/target/native

##### Monitoring Configuration Settings #####

# Actor configuration

cinnamon.akka {
  actors {
    "samle.cluster.*" {
      report-by = class
      traceable = on
    }
    "/user/*" {
      report-by = class
      traceable = on
    }
  }
}

Note that in cinnamon.akka.actors settings set traceable = on enables actor tracing. That’s it! Of course, there are other configuration settings available, but this is enough to trace actor activity. At this point we should be able to run the project and look at some tracing data.

Tracing Akka Actors

Now that sbt and Akka are configured it is time to run some things to view some tracing data. If you are following along using the akka-samples-cluster-java project, please review the README file. This is an interesting project that showcases various Akka clustering features. For this initial tracing test, we will run the Router Example with Group of Routees processes.

Before we run anything, we are first going to start up Jaeger and Zipkin so that we can view the trace data in their respective UIs. Fortunately, both Jaeger and Zipkin provide Docker images that make it easy to use these components. The documentation provides all of the details. Here we cut to the chase and just run the two Docker instances.

docker run -d -p5775:5775/udp -p16686:16686 jaegertracing/all-in-one:latest

docker run -d -p 9411:9411 openzipkin/zipkin

The docker ps command shows the running instances and the tcp ports for their UIs.

$ docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                                                                       NAMES
717712f95641        openzipkin/zipkin                 "/bin/sh -c 'test ..."   26 hours ago        Up 4 seconds        9410/tcp, 0.0.0.0:9411->9411/tcp                                            gallant_payne
a46c70f06192        jaegertracing/all-in-one:latest   "/go/bin/standalon..."   28 hours ago        Up 28 hours         0.0.0.0:5775->5775/udp, 5778/tcp, 6831-6832/udp, 0.0.0.0:16686->16686/tcp   awesome_kirch

Note the Zipkin TCP port is 9411 and the Jaeger TCP port is 16686. Access the Jaeger UI at http://localhost:16686 and the Zipkin UI at http://localhost:9411. There will not be anything to see until we run something so let’s do that next.

The akka-samples-cluster-java project README file provides instructions for running various Akka clustering examples. We are going to run some tests with Router Example with Group of Routees. There are two approaches for running the cluster. You can either run one program that starts up the cluster or start each Akka cluster node separately. Since we are mainly interested in tracing Akka actors, it does not matter how you run the cluster. That said, it is more interesting to run each cluster node separately in different terminal windows.

To run everything with a single command cd into the project directory and enter the command:

sbt "runMain sample.cluster.stats.StatsSampleMain"

To run each cluster node separately open four terminal windows, in each terminal window cd into the project directory and enter the following commands, one in each terminal window:

sbt "runMain sample.cluster.stats.StatsSampleMain 2551"
sbt "runMain sample.cluster.stats.StatsSampleMain 2552"
sbt "runMain sample.cluster.stats.StatsSampleClientMain"
sbt "runMain sample.cluster.stats.StatsSampleMain 0"

Once the Akka cluster has been running for a few minutes go take a look at the Jaeger UI and Zipkin UI. Each UI is similar. First select the service, which in this case defaults to the main class name.

image alt text

Jager UI Example.

image alt text

ZipKin UI Example.

Shown above are screenshots of the Jaeger UI and the Zipkin UI.

So that is it! A few changes to the build configuration plus a few changes to the Akka configuration and Akka actor tracing is enabled. No code changes. No annotations. Just configuration changes.

At this point, you are ready to explore the tracing data via either or both UIs. There are also other configuration settings available that you can use to fine tune the monitoring settings. Give it a try and learn more about how you can use this very useful and new monitoring telemetry feature.

In future articles, we will explore tracing and telemetry in more detail. We will also explore monitoring with the exciting new advanced monitoring solution OpsClarity.


Your subscription to the Lightbend Reactive Platform includes enterprise capabilities that extend functionality beyond what’s available in the open source projects to deliver an enterprise-grade platform ready for your mission-critical systems. Learn more about how to use the Monitoring commercial features every month with our new Monitoring Tip Of The Month.

If you have general questions about how to utilize your subscription better, please contact support. If you have additional questions, or need assistance with Monitoring open a support ticket or email support@lightbend.com, and we will be happy to help.