OpenTracing configuration

Entities such as actors, streams, or HTTP endpoints need to be configured for tracing. There are also various options around tracing, such as trace sampling, naming, or tagging.

Actor configuration

Actors need to be enabled for tracing, similar to metrics and events. This is an extension of the actor configuration, with a traceable setting that can be enabled for any actor selection.

For example, actors can be selected by class or path and then enabled as traceable, such as in the following configuration:

cinnamon.akka {
  actors {
    "com.example.a.b.*" {
      report-by = class
      traceable = on
    }
    "/user/x/*" {
      report-by = class
      traceable = on
    }
  }
}

System messages

Akka system messages (special internal messages for managing actors) can also be traced. This is off by default, but can be enabled with this configuration:

cinnamon.opentracing {
  akka {
    trace-system-messages = on
  }
}

Akka Scheduler configuration

Context propagation

By default, OpenTracing context will be propagated with scheduleOnce.

Note: If you use scheduleOnce in a way that it forms cycles it will create potentially infinite traces. In order to break such infinite traces disable the connect-traces setting.

Span creation

By default, no spans will be created for the runnable execution time passed to scheduleOnce. Whether to build spans or not also depends on the Auto trace setting. Here is the decision table:

Auto trace The build-spans setting Build spans Comment
off off no By default no spans will be built.
off on yes The build-spans setting enables span creation.
on on/off yes Auto trace enables span creation for active traces regardless of build-spans setting.
Reference
cinnamon.opentracing {
  akka.scheduler {
    schedule-once {
      # Whether to connect traces for asynchronous calls scheduled with the Akka Scheduler scheduleOnce.
      # Turn it off to disconnect possibly infinite traces.
      connect-traces = on
      # Whether to build spans for the execution of the Runnable passed to the Akka Scheduler scheduleOnce.
      # Note that it will create spans even if build-spans is off when auto-trace is enabled and it's an active trace.
      build-spans = off
    }
  }
}

Akka Streams configuration

Akka Streams need to be enabled for tracing, similar to metrics. This is an extension of the Akka Stream configuration, with a traceable setting that can be enabled when configuring streams for instrumentation.

For example, streams can be selected using configuration based on the materialization call site and then enabled as traceable, such as in the following config:

cinnamon.akka {
  streams {
    "example.streams.a.A.*" {
      report-by = name
      traceable = on
    }
  }
}

Or streams can be enabled as traceable using the Instrumented attribute, such as in the following example:

Scala
import com.lightbend.cinnamon.akka.stream.CinnamonAttributes._

Source.single(0)
  .map(_ + 1)
  .to(Sink.ignore)
  .instrumented(name = "my-stream", traceable = true)
  .run()
Java
import com.lightbend.cinnamon.akka.stream.CinnamonAttributes;

Source.single(0)
    .map(x -> x + 1)
    .to(Sink.ignore())
    .addAttributes(
        CinnamonAttributes.isInstrumented()
            .withReportByName("my-stream")
            .setTraceable()
            .attributes())
    .run(materializer);

Akka HTTP configuration

Akka HTTP endpoints need to be enabled for tracing, similar to metrics. This is an extension of the Akka HTTP configuration, with a traceable setting that can be enabled for any endpoint selection.

For example, endpoint paths can be selected using a wildcard and then enabled as traceable, such as in the following configuration:

cinnamon.akka.http {
  clients {
    "*:*" {
      paths {
        "*" {
          metrics = on
          traceable = on
        }
      }
    }
  }
  servers {
    "*:*" {
      paths {
        "*" {
          metrics = on
          traceable = on
        }
      }
    }
  }
}

Akka HTTP internal actors

An Akka HTTP server also creates actors under the /user guardian. If you have enabled actor tracing with a /user/* selection, then internal Akka HTTP and Akka Streams actors will also appear in traces. You can select actors by package instead, to only trace application actors. Or you can exclude the internal actor packages from a /user/* selection, such as in the following configuration:

cinnamon.akka {
  actors {
    "/user/*" {
      excludes = ["akka.http.*", "akka.stream.*"]
      report-by = class
      traceable = on
    }
  }
}

Akka HTTP span context propagation

If tracing is enabled, Lightbend Telemetry will automatically handle propagation of existing span contexts. For Akka HTTP, tracing means that the context is extracted from a trace span and injected into the headers of an outgoing HTTP Request (client-side), and extracting the context from an incoming HTTP Request to create a span (server-side).

Below is an image of what a cross-service trace can look like when using the trace propagation feature:

Play configuration

Play endpoints need to be enabled for tracing, similar to metrics.

For example, endpoint paths can be selected using a wildcard and then enabled as traceable, such as in the following configuration:

cinnamon.play.http {
  clients {
    "*:*" {
      paths {
        "*" {
          metrics = on
          traceable = on
        }
      }
    }
  }
  servers {
    "*:*" {
      paths {
        "*" {
          metrics = on
          traceable = on
        }
      }
    }
  }
}

Play span context propagation

If tracing is enabled, Lightbend Telemetry will automatically handle propagation of existing span contexts. For Play, tracing means that the context is extracted from a trace span and injected into the headers of an outgoing HTTP Request (client-side), and extracting the context from an incoming HTTP Request to create a span (server-side).

Lagom configuration

Lagom endpoints need to be enabled for tracing, similar to metrics.

For example, endpoint paths can be selected using a wildcard and then enabled as traceable, such as in the following configuration:

cinnamon.lagom.http {
  clients {
    "*:*" {
      paths {
        "*" {
          metrics = on
          traceable = on
        }
      }
    }
  }
  servers {
    "*:*" {
      paths {
        "*" {
          metrics = on
          traceable = on
        }
      }
    }
  }
}

Scala Futures

Active traces will be automatically propagated through Scala Futures, but scheduled Futures or callbacks will not be automatically represented as trace spans. To enable tracing of Futures, there is a naming API to indicate Futures or Future callbacks that should be traced and to specify the trace span operation name.

For example, there is a named alternative to Future.apply which allows scheduled Futures to be traced:

Scala
// this Future is not traceable
val future = Future {
  "compute all the things"
}

import com.lightbend.cinnamon.scala.future.named._

// this Future is traceable and named "compute"
val tracedFuture = FutureNamed("compute") {
  "compute all the things"
}

There are also named alternatives for the callback operations, which are added implicitly as extension methods on Future. For example, to name and trace a mapped transform operation between actors, the mapNamed method can be used in place of map:

Scala
import com.lightbend.cinnamon.scala.future.named._

val foo = tracedActor("foo")
val bar = tracedActor("bar")

val future = foo ? message

val transformed = future.mapNamed("transform") {
  value => transform(value)
}

transformed pipeTo bar

This transformation will then show up as its own trace span, between the actor spans, such as in this trace:

Cached Scala Futures

Cinnamon tracks two contexts for Scala Futures: the context where callbacks are added, and the context where the underlying Promise is completed. For tracing, the completion context is preferred so that traces follow the dataflow. If Futures are cached and then reused in subsequent traces, any callbacks will be connected with the original trace context. In this case, the completion context can be disabled for a Future by using the disableContext method included in the nameable Future extensions:

Scala
import com.lightbend.cinnamon.scala.future.named._

traceSpan("one") {
  // future is created within the first trace context
  val future = FutureNamed("compute") {
    "compute all the things"
  }
  // the future is cached to reuse the result
  // but will not connect to this first trace context
  cached += "foo" -> future.disableContext
}

traceSpan("two") {
  // this second trace context will be used for future callbacks
  cached("foo").mapNamed("something")(doSomething)
}

Java Futures

Active traces will be automatically propagated through Java CompletableFutures, but scheduled CompletableFutures or CompletionStage callbacks will not be automatically represented as trace spans. To enable tracing of CompletableFutures, there is a naming API to indicate CompletableFutures or CompletionStage callbacks that should be traced and to specify the trace span operation name.

For example, NameableCompletableFuture has a named alternative to CompletableFuture.supplyAsync which allows scheduled CompletableFutures to be traced:

Java
import com.lightbend.cinnamon.java.future.NameableCompletableFuture;
import java.util.concurrent.CompletableFuture;

// this CompletableFuture is not traceable
CompletableFuture<String> future =
    CompletableFuture.supplyAsync(() -> "compute all the things");

// this CompletableFuture is traceable and named "compute"
CompletableFuture<String> tracedFuture =
    NameableCompletableFuture.supplyAsyncNamed("compute", () -> "compute all the things");

There is also a NameableCompletionStage with named alternatives for the CompletionStage callback operations. For example, to name and trace a transform operation between actors the thenApplyNamed method can be used in place of thenApply:

Java
import akka.actor.ActorRef;
import akka.pattern.Patterns;
import java.util.concurrent.CompletionStage;

import static com.lightbend.cinnamon.java.future.NameableCompletionStage.nameable;

final ActorRef foo = tracedActor.named("foo");
final ActorRef bar = tracedActor.named("bar");

CompletionStage<Object> future = Patterns.ask(foo, message, timeout);

CompletionStage<Object> transformed =
    nameable(future).thenApplyNamed("transform", value -> transform.apply(value));

Patterns.pipe(transformed, system.dispatcher()).to(bar, sender);

This transformation will then show up as its own trace span, between the actor spans, such as in this trace:

Cached Java Futures

Cinnamon tracks two contexts for Java CompletableFutures: the context where callbacks are added, and the context where the future is completed. For tracing, the completion context is preferred so that traces follow the dataflow. If CompletionStages are cached and then reused in subsequent traces, any callbacks will be connected with the original trace context. In this case, the completion context can be disabled for a CompletionStage or CompletableFuture by using the disableContext method included in the nameable extensions:

Java
import com.lightbend.cinnamon.java.future.NameableCompletableFuture;
import com.lightbend.cinnamon.java.future.NameableCompletionStage;

import static com.lightbend.cinnamon.java.future.NameableCompletionStage.nameable;

traceSpan(
    "one",
    () -> {
      // future is created within the first trace context
      CompletionStage<String> future =
          NameableCompletableFuture.supplyAsyncNamed("compute", () -> "compute all the things");
      // the future is cached to reuse the result
      // but will not connect to this first trace context
      cached.put("foo", NameableCompletionStage.disableContext(future));
    });

traceSpan(
    "two",
    () -> {
      // this second trace context will be used for future callbacks
      nameable(cached.get("foo")).thenApplyNamed("something", value -> doSomething(value));
    });

Tracing configuration

The OpenTracing integration for both Jaeger and Zipkin build on the [Jaeger client]. The tracer supports the following configuration:

Setting a service name for each node is useful. The service name can be configured specifically for tracing using the service-name setting (example below) or otherwise this will be based on the application name from the shared Cinnamon metadata. You can use the cinnamon.application setting to configure the same name for both metrics and tracing.

Global tags can be added to the tracer, which will be added to all trace spans. See trace span tags for adding tags to specific types of spans.

Note: Tracing can produce a very high volume of data, so sampling is applied (at the beginning of a trace). The sampler used, and its settings, can be configured. The default sampler is a rate-limiting sampler that captures up to 10 traces per second.

On the Example tab, there is a configuration that sets the service-name to my-component, sets a custom environment tag to staging, and configures a rate-limiting sampler with a maximum of 25 traces per second:

Required

There is nothing to configure if you want to use the default OpenTracing settings that will use the rate limiting sampler with 10 traces per second.

Example
cinnamon.opentracing {
  tracer {
    service-name = "my-component"

    tags {
      environment = "staging"
    }

    sampler = rate-limiting-sampler

    rate-limiting-sampler {
      max-traces-per-second = 25
    }
  }
}
Reference
cinnamon.opentracing {
  tracer {

    # Service name for this application, defaults to the `cinnamon.application` identifier when not set
    service-name = null

    # Tags added to all trace spans (key:value pairs)
    tags {}

    # Trace sampler to use
    sampler = rate-limiting-sampler

    rate-limiting-sampler {
      # Maximum number of sampled traces per second
      max-traces-per-second = 10
    }

    probabilistic-sampler {
      # Probabilistic sampling rate, between 0.0 and 1.0
      sampling-rate = 0.001
    }

    const-sampler {
      # Constant decision on whether to sample traces
      # Note: this sampler is NOT recommended for production
      decision = true
    }

    # Propagation codecs for cross-process tracing (multiple codecs can be active)
    # Include B3 propgation with: `cinnamon.opentracing.tracer.propagations += b3-propagation`
    propagations = [jaeger-propagation]

    jaeger-propagation {
      # Whether to URL encode the trace context for HTTP header propagation
      http-header-url-encoding = on
    }

    b3-propagation {
    }

    # Log trace spans with SLF4J (can be used for debugging the tracer)
    # Set `cinnamon.opentracing.tracer.reporters += trace-logging`
    trace-logging {
      # Name of SLF4J logger to use when logging
      logger = "cinnamon.opentracing.Tracer"
    }

  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Trace span names

By default, Cinnamon creates trace span names with two parts, the operation and the resource or entity being traced:

<operation>: <resource>

Here are some example span names for an Akka HTTP request, an Akka Actor message receive, and an Akka Stream operator:

"akka.http.server.request: GET /hello/<String>"
"akka.actor.receive: /user/actor"
"akka.stream.operator: my-stream-0-1-map"

To exclude the resource name from trace span names and only have the static operation name, you can disable the following configuration setting:

cinnamon.opentracing {
  span.name {
    include-resource = off
  }
}

You can also modify the operation names used for trace spans via configuration. For example, to change the operation name used for actor receive spans, you can use the following configuration:

Example
cinnamon.opentracing {
  spans {
    akka.actor.receive.operation = "actor"
  }
}
Reference
cinnamon.opentracing {
  span.name {
    # Whether to include resource names in full span names
    include-resource = on
  }

  # Settings for trace span building
  spans {
    akka.actor {
      receive {
        operation = "akka.actor.receive"
        tags {
          component = "akka.actor"
          span.kind = "consumer"
        }
      }
      system-message {
        operation = "akka.actor.system-message"
        tags {
          component = "akka.actor"
          span.kind = "consumer"
        }
      }
    }

    akka.scheduler {
      schedule-once {
        operation = "akka.scheduler.schedule-once"
        tags {
          component = "akka.scheduler"
          span.kind = "consumer"
        }
      }
    }

    akka.stream {
      operator {
        operation = "akka.stream.operator"
        tags {
          component = "akka.stream"
          span.kind = "consumer"
        }
      }
      boundary {
        operation = "akka.stream.boundary"
        tags {
          component = "akka.stream"
          span.kind = "consumer"
        }
      }
      buffer {
        operation = "akka.stream.buffer"
        tags {
          component = "akka.stream"
          span.kind = "consumer"
        }
      }
    }

    akka.http {
      client {
        operation = "akka.http.client.request"
        tags {
          component = "akka.http"
          span.kind = "client"
        }
      }
      server {
        operation = "akka.http.server.request"
        tags {
          component = "akka.http"
          span.kind = "server"
        }
      }
    }

    play.ws {
      client {
        operation = "play.ws.client.request"
        tags {
          component = "play.ws"
          span.kind = "client"
        }
      }
    }
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Trace span tags

Cinnamon will automatically include tags for trace spans, both static tags—that are always the same for a particular type of span—and dynamic tags—which are based on the traced entity, or requests, messages, or elements received.

Static tags

The static tags for a trace span can be configured, and extra tags can be added via configuration. For example, to change the provided component tag for an Akka HTTP server request, and to add a new tag named span.type set to web, we can use the following configuration:

Example
cinnamon.opentracing {
  spans {
    akka.http.server {
      tags {
        component = "akka.http.server"
        span.type = "web"
      }
    }
  }
}
Reference
cinnamon.opentracing {
  span.name {
    # Whether to include resource names in full span names
    include-resource = on
  }

  # Settings for trace span building
  spans {
    akka.actor {
      receive {
        operation = "akka.actor.receive"
        tags {
          component = "akka.actor"
          span.kind = "consumer"
        }
      }
      system-message {
        operation = "akka.actor.system-message"
        tags {
          component = "akka.actor"
          span.kind = "consumer"
        }
      }
    }

    akka.scheduler {
      schedule-once {
        operation = "akka.scheduler.schedule-once"
        tags {
          component = "akka.scheduler"
          span.kind = "consumer"
        }
      }
    }

    akka.stream {
      operator {
        operation = "akka.stream.operator"
        tags {
          component = "akka.stream"
          span.kind = "consumer"
        }
      }
      boundary {
        operation = "akka.stream.boundary"
        tags {
          component = "akka.stream"
          span.kind = "consumer"
        }
      }
      buffer {
        operation = "akka.stream.buffer"
        tags {
          component = "akka.stream"
          span.kind = "consumer"
        }
      }
    }

    akka.http {
      client {
        operation = "akka.http.client.request"
        tags {
          component = "akka.http"
          span.kind = "client"
        }
      }
      server {
        operation = "akka.http.server.request"
        tags {
          component = "akka.http"
          span.kind = "server"
        }
      }
    }

    play.ws {
      client {
        operation = "play.ws.client.request"
        tags {
          component = "play.ws"
          span.kind = "client"
        }
      }
    }
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

String representations

Cinnamon will automatically include tags in a trace span for objects like the message received by an actor or the element processed by a stream operator. By default, the toString of the object is used, but this behavior is configurable. To switch to using just the class name for these objects, use the following configuration:

cinnamon.opentracing {
  object-formatter = object-to-class-name
}

Custom object formatter

A custom object formatter, for string representations of objects in trace spans, can also be configured. An ObjectFormatter class needs to be implemented and configuration added to use this formatter.

For example, we can create a CustomObjectFormatter class, implementing the formatToString method:

import com.lightbend.cinnamon.opentracing.ObjectFormatter

class CustomObjectFormatter extends ObjectFormatter {
  override def formatToString(someObject: Object): String = {
    String.valueOf(someObject).toUpperCase
  }
}

We can then configure the OpenTracing integration to use this formatter with:

cinnamon.opentracing {
  object-formatter = custom-object-formatter
  custom-object-formatter {
    formatter-class = "opentracing.api.sample.CustomObjectFormatter"
  }
}

Debug traces (forced sampling)

Traces that begin with an HTTP request can have a “debug mode” enabled using an HTTP header. When a trace debug header is configured and present on an HTTP request, then the request will have a sampling priority tag added, to hint that this trace should be force sampled. For the default tracer, used for Jaeger and Zipkin reporters, the debug flag will also be enabled—which flags the trace to survive all downsampling that might happen in the trace collection pipeline. The value of the trace debug header can be used as a correlation identifier as it will be added as a trace tag.

To enable debug traces, set the name of the debug header in configuration:

Example
cinnamon.opentracing {
  http {
    debug-header = "Trace-Debug"
  }
}

This header can then be set on HTTP requests to force sample traces and provide a correlation identifier for the trace:

Example
"Trace-Debug" -> "some-correlation-id"

When using the default tracer (based on the Jaeger client) then the built-in debug mode via HTTP headers can also be used—by using the jaeger-debug-id header rather than configuring a custom header.

Auto trace

Cinnamon will only create trace spans for entities that have been enabled for tracing. To connect together traces across any intermediary asynchronous boundaries which are not enabled for tracing, Cinnamon will use a “pass-through” mode—where the trace context is passed through without any trace spans being created. It’s also possible to enable “auto tracing” which will automatically create trace spans for the intermediate steps, whenever these occur within a currently active and sampled trace. Note that enabling auto traces can increase the number of trace spans significantly, and will also trace both user code and framework internals. To enable auto traces, use this configuration setting:

cinnamon.opentracing {
  auto-trace = on
}

To only enable auto traces when within a debug trace use the following configuration instead:

cinnamon.opentracing {
  auto-trace-when-debug = on
}