Cinnamon OpenTracing APIs

The OpenTracing integration includes APIs for attaching additional information to active trace spans, and works alongside direct use of the OpenTracing API.

Active span

OpenTracing supports recording logs to a trace span and also attaching baggage to the trace context—key:value string pairs which are propagated with the trace, similar to a logging MDC. Cinnamon provides access to the currently active span and there are utility methods for logging and attaching baggage to this span.

Span logs

Cinnamon includes utility methods for logging events or structured data to the currently active span.

The Cinnamon ActiveSpan API can be imported with:

import com.lightbend.cinnamon.opentracing.ActiveSpan

You can log an event to the active span:

ActiveSpan.log("something")

You can log structured data (a Java Map) to the active span:

ActiveSpan.log(ImmutableMap.of("a", "one", "b", "two"))

Trace baggage

Cinnamon includes utility methods for attaching baggage to the trace context—key:value string pairs which are propagated with the trace.

The Cinnamon ActiveSpan API can be imported with:

import com.lightbend.cinnamon.opentracing.ActiveSpan

A baggage item (a key:value string pair) can be attached to the current trace:

ActiveSpan.setBaggageItem("token", "abc123")

Baggage items can also be accessed from anywhere deeper in a trace:

ActiveSpan.getBaggageItem("token")

Note: Baggage keys are case-insensitive. This is aligned with case-insensitive field names in HTTP headers.

Note: Baggage items are transferred throughout the trace, both locally and remotely, which can introduce some extra overhead.

GlobalTracer active span

You can also access the currently active span through the GlobalTracer available in the OpenTracing Java API, and then use the OpenTracing API directly to record logs or attach baggage. For example, you can log to the active span via the global tracer:

import io.opentracing.util.GlobalTracer

GlobalTracer.get.activeSpan.log("something")

Custom spans

You can add custom trace spans using the OpenTracing Java API. Cinnamon tracing is integrated with the GlobalTracer and active spans managed by the ThreadLocalScopeManager, both provided by the opentracing-util module.

The global tracer can be accessed with:

import io.opentracing.util.GlobalTracer

val tracer = GlobalTracer.get

You can create a custom span using the OpenTracing API:

val scope = tracer.buildSpan("custom-span").startActive( /*finishOnClose =*/ true)
// do some work within the scope of this active span ...
scope.close()

If there is anything asynchronous within the scope of a custom span, and which is instrumented by Cinnamon, then traces will be automatically connected to the custom spans.

It’s also possible to manually propagate traces across process boundaries using the OpenTracing APIs for injecting and extracting trace contexts.

Note that to create custom spans that will connect to Akka Stream operators, you need to scope over the actual downstream push to a stage, which requires a custom stream stage. As an example, if there was a message service (not instrumented by Cinnamon automatically) where the consumer was an Akka Stream, it’s possible to use the OpenTracing APIs to inject and extract trace contexts into any messages that support headers, and to scope any downstream operators with a custom trace span using a custom stream stage. Here’s a complete example of doing this:

import akka.stream._
import akka.stream.scaladsl._
import akka.stream.stage.{ GraphStage, GraphStageLogic, InHandler, OutHandler }
import com.lightbend.cinnamon.akka.stream.CinnamonAttributes._
import io.opentracing.propagation.{ Format, TextMapExtractAdapter, TextMapInjectAdapter }
import io.opentracing.util.GlobalTracer
import io.opentracing.{ Scope, SpanContext, Tracer }
import java.util.{ HashMap => JHashMap, Map => JMap }
import scala.collection.JavaConverters._

// demo message type with headers for context
case class Message[T](headers: Map[String, String], payload: T)

// access the global tracer
val tracer: Tracer = GlobalTracer.get

// start trace span on the producer side
val producerScope: Scope = tracer
  .buildSpan("producer")
  .startActive( /*finishOnClose =*/ true)

// access the currently active context when sending a message
val producerContext: SpanContext = tracer.activeSpan().context()

// inject the headers for the parent context into a text map
val contextHeaders: JMap[String, String] = new JHashMap[String, String]()
tracer.inject(
  producerContext,
  Format.Builtin.TEXT_MAP,
  new TextMapInjectAdapter(contextHeaders))

// store the trace headers in the message
val message = Message(contextHeaders.asScala.toMap, "some payload")

// close the trace scope for the producer (finish and deactivate the trace span)
producerScope.close()

// custom stream stage so we can wrap the downstream `push` of the message payload with traced scope
class Extract[T] extends GraphStage[FlowShape[Message[T], T]] {
  val in = Inlet[Message[T]]("extract.in")
  val out = Outlet[T]("extract.out")
  override val shape = FlowShape(in, out)

  override def initialAttributes: Attributes = Attributes.name("extract")

  override def createLogic(inheritedAttributes: Attributes): GraphStageLogic =
    new GraphStageLogic(shape) with InHandler with OutHandler {
      override def onPush(): Unit = {
        val message = grab(in)
        // extract the trace context from the message headers
        val context: SpanContext = tracer.extract(
          Format.Builtin.TEXT_MAP,
          new TextMapExtractAdapter(message.headers.asJava))
        // use the context as a parent reference for a consumer trace span
        val consumerScope: Scope = tracer
          .buildSpan("consumer")
          .asChildOf(context)
          .startActive( /*finishOnClose =*/ true)
        // push the message payload downstream scoped by the connected trace span
        push(out, message.payload)
        // close the trace scope for the consumer (finish and deactivate the trace span)
        consumerScope.close()
      }

      override def onPull(): Unit = pull(in)

      setHandlers(in, out, this)
    }
}

implicit val materializer = ActorMaterializer()

// imagine the message is actually transferred via some message service...
Source.single(message)
  .via(new Extract[String])
  .map(_.toUpperCase)
  .to(Sink.ignore)
  .instrumented(name = "sample", traceable = true)
  .run()