OpenTracing backends

The OpenTracing integration can be used with any OpenTracing-compatible tracing solution. There is included integration with Jaeger and Zipkin.

Jaeger reporter

Jaeger is a distributed tracing system with support for OpenTracing.

Cinnamon Jaeger dependency

First make sure that your build is configured to use the Cinnamon Agent.

To enable the Jaeger reporter, add the following dependencies to your build:

sbt

libraryDependencies += Cinnamon.library.cinnamonOpenTracing
libraryDependencies += Cinnamon.library.cinnamonOpenTracingJaeger

Maven

<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-opentracing_2.12</artifactId>
  <version>2.11.3</version>
</dependency>
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-opentracing-jaeger</artifactId>
  <version>2.11.3</version>
</dependency>

Gradle

dependencies {
  compile(
    [group: 'com.lightbend.cinnamon', name: 'cinnamon-opentracing_2.12', version: '2.11.3'],
    [group: 'com.lightbend.cinnamon', name: 'cinnamon-opentracing-jaeger', version: '2.11.3'],
  )
}

Jaeger configuration

Jaeger reporting can be configured. On the Example tab, there is a configuration that sets a different endpoint for the Jaeger agent by configuring the host and port settings:

Required

There is nothing to configure if you want to use the default Jaeger settings that will communicate with localhost on port 5775.

Example

cinnamon.opentracing {
  jaeger {
    host = "localhost"
    port = 5432
  }
}

Reference

cinnamon.opentracing {
  jaeger {

    # Host for Jaeger trace span collector
    host = "localhost"

    # UDP port for Jaeger trace span collector
    port = 5775

    # Max size for UDP packets
    max-packet-size = 65000

    # Flush interval for trace span reporter
    flush-interval = 1s

    # Max queue size of trace span reporter
    max-queue-size = 1000

  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Running Jaeger

See the Jaeger documentation for running Jaeger. The Jaeger getting started shows how to run Jaeger locally for development and testing.

Here’s what an example actor trace in Jaeger looks like:

Zipkin reporter

Zipkin is a distributed tracing system with support for OpenTracing.

Cinnamon Zipkin dependency

First make sure that your build is configured to use the Cinnamon Agent.

To enable the Zipkin reporter, add the following dependencies to your build:

sbt

libraryDependencies += Cinnamon.library.cinnamonOpenTracing
libraryDependencies += Cinnamon.library.cinnamonOpenTracingZipkin

Maven

<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-opentracing_2.12</artifactId>
  <version>2.11.3</version>
</dependency>
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-opentracing-zipkin</artifactId>
  <version>2.11.3</version>
</dependency>

Gradle

dependencies {
  compile(
    [group: 'com.lightbend.cinnamon', name: 'cinnamon-opentracing_2.12', version: '2.11.3'],
    [group: 'com.lightbend.cinnamon', name: 'cinnamon-opentracing-zipkin', version: '2.11.3']
  )
}

Zipkin configuration

The default Zipkin sender is the URL connection sender, which can be used for sending trace spans directly to the Zipkin API. This sender can be configured. On the Example tab there is a configuration that sets a different endpoint for the Zipkin trace span collector by configuring the endpoint setting:

Required

There is nothing to configure if you want to use the default Zipkin settings that will communicate with localhost on port 9411 using the URL connection sender.

Example

cinnamon.opentracing {
  zipkin {
    url-connection {
      endpoint = "http://my.zipkin.host:9411/api/v2/spans"
      encoding = "proto3"
    }
  }
}

Reference

cinnamon.opentracing {
  zipkin {

    # Zipkin API Version (1 or 2)
    # It is used to select a proper version of JSON encoding (JSON_V1 or JSON_V2)
    # and to inform about a possible inconsistency between Zipkin API version and:
    #  - supported encoding format or
    #  - the endpoint URL when used with the `url-connection` sender.
    api-version = 2

    # How long to block for in-flight spans to send out-of-process on close.
    close-timeout = 1s

    # Instead of sending one message at a time, spans are bundled into messages, up to `message-max-bytes`.
    # This timeout ensures that spans are not stuck in an incomplete message.
    # Note: this timeout starts when the first unsent span is reported.
    message-timeout = 1s

    # Flush interval for trace span reporter. Must be greater than zero.
    # DEPRECATED! This is an obsolete setting. Use `message-timeout` instead.
    flush-interval = null

    # Maximum backlog of spans reported vs sent.
    queued-max-spans = 10000

    # Max queue size of trace span reporter.
    # Maximum backlog of spans reported vs sent.
    # DEPRECATED! This is an obsolete setting. Use `queued-max-spans` instead.
    max-queue-size = null

    # Maximum backlog of span bytes reported vs sent. Default 1% of heap.
    queued-max-bytes = null

    # Zipkin sender to use for reporting trace spans
    sender = url-connection

    # Maximum bytes sendable per message including overhead.
    # Defaults to, and is limited by `sender.message-max-bytes`.
    message-max-bytes = null

    # URL connection sender for reporting directly to a Zipkin API endpoint
    url-connection {

      # Zipkin API POST URL
      endpoint = "http://localhost:9411/api/v2/spans"

      # Encoding to use for trace spans:
      #   proto3 - requires Zipkin 2.8+, endpoint should point to `api/v2`
      #   json   - default encoding to make it work with Zipkin prior 2.8
      #            when using with Zipkin 2, `endpoint` should point to `api/v2` and `api-version` should be set to 2
      #   thrift - DEPRECATED and will be removed in future versions!
      encoding = "proto3"

      # Timeout for establishing URL connection
      connect-timeout = 10s

      # Timeout for connection reads
      read-timeout = 60s

      # Whether GZIP compression is enabled
      compression = true

      # Maximum bytes sendable per message including overhead.
      message-max-bytes = 5MiB
    }

  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

See the following sections for configuring the Zipkin sender for Kafka or Scribe.

Zipkin Kafka sender

Zipkin can be configured to send traces to a Kafka topic. This sender supports Kafka 0.10.2+.

To enable the Zipkin Kafka sender, add the following dependencies to your build:

sbt

libraryDependencies += Cinnamon.library.cinnamonOpenTracing
libraryDependencies += Cinnamon.library.cinnamonOpenTracingZipkinKafka

Maven

<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-opentracing_2.12</artifactId>
  <version>2.11.3</version>
</dependency>
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-opentracing-zipkin-kafka</artifactId>
  <version>2.11.3</version>
</dependency>

Gradle

dependencies {
  compile(
    [group: 'com.lightbend.cinnamon', name: 'cinnamon-opentracing_2.12', version: '2.11.3'],
    [group: 'com.lightbend.cinnamon', name: 'cinnamon-opentracing-zipkin-kafka', version: '2.11.3']
  )
}

You can then configure the Zipkin reporter to use the Kafka sender. You must specify the Kafka bootstrap servers to use. You can also override any of the producer configs using the properties configuration section:

Required

cinnamon.opentracing {
  zipkin {
    sender = kafka

    kafka {
      bootstrap-servers = ["my.kafka.host1:9091", "my.kafka.host2:9091"]
      encoding = "proto3"
    }
  }
}

Reference

cinnamon.opentracing {
  zipkin {
    kafka {
      # Initial set of kafka servers to connect to (must be specified)
      bootstrap-servers = []

      # Kafka topic to send trace spans to
      topic = "zipkin"

      # Encoding to use for trace spans:
      #   json   - default encoding to make it work with Zipkin prior 2.8
      #   proto3 - requires Zipkin 2.8+
      #   thrift - DEPRECATED and will be removed in future versions!
      encoding = "json"

      # Property overrides for producer configs (http://kafka.apache.org/0102/documentation.html#producerconfigs)
      properties {}

      # Maximum size of messages
      max-message-size = 1MB
    }
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Zipkin Scribe sender

Zipkin can be configured to send traces to Scribe.

To enable the Zipkin Scribe sender, add the following dependencies to your build:

sbt

libraryDependencies += Cinnamon.library.cinnamonOpenTracing
libraryDependencies += Cinnamon.library.cinnamonOpenTracingZipkinScribe

Maven

<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-opentracing_2.12</artifactId>
  <version>2.11.3</version>
</dependency>
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-opentracing-zipkin-scribe</artifactId>
  <version>2.11.3</version>
</dependency>

Gradle

dependencies {
  compile(
    [group: 'com.lightbend.cinnamon', name: 'cinnamon-opentracing_2.12', version: '2.11.3'],
    [group: 'com.lightbend.cinnamon', name: 'cinnamon-opentracing-zipkin-scribe', version: '2.11.3']
  )
}

You can then configure the Zipkin reporter to use the Scribe sender. On the Example tab there is a configuration that changes the Scribe endpoint using the host and port settings:

Required

cinnamon.opentracing {
  zipkin {
    sender = scribe
  }
}

Example

cinnamon.opentracing {
  zipkin {
    sender = scribe

    scribe {
      host = "my.scribe.host"
      port = 9410
    }
  }
}

Reference

cinnamon.opentracing {
  zipkin {
    scribe {
      # Host of Scribe trace collector
      host = "localhost"

      # Port of Scribe trace collector
      port = 9410

      # Timeout for socket reads
      socket-timeout = 60s

      # Timeout for connections
      connect-timeout = 10s

      # Maximum size of messages (scribe default is 16384000 bytes)
      max-message-size = 16000KiB
    }
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Running Zipkin

See the Zipkin documentation for running Zipkin. The Zipkin quickstart shows how to run Zipkin locally for development and testing.

Here’s what an example actor trace in Zipkin looks like:

Zipkin B3 propagation

Zipkin-compatible tracing systems usually propagate the trace context using B3 headers, such as X-B3-TraceId. To integrate with other HTTP services that do not use Cinnamon or Jaeger-based tracing but do have Zipkin integration, the Tracer can be configured to inject and extract the trace context using B3 headers.

You can configure the tracer to use B3 propagation, in addition to the default Jaeger propagation:

cinnamon.opentracing.tracer {
  propagations += b3-propagation
}

Custom OpenTracing tracers

It’s possible to create OpenTracing compatible tracers programmatically, by providing a Cinnamon TracerFactory that creates the Tracer directly.

For example, the LightStep Tracer can be used by implementing a TracerFactory such as:

import com.lightbend.cinnamon.logging.LoggingProvider
import com.lightbend.cinnamon.opentracing.TracerFactory
import com.typesafe.config.Config
import io.opentracing.Tracer

/**
 * Optional parameters for an implementation of a TracerFactory (in this specific order):
 *
 * @param configPath      path to section in config for the tracer, e.g. `cinnamon.opentracing.lightstep`
 * @param config          the configuration for this `ActorSystem`
 * @param loggingProvider a Cinnamon `LoggingProvider`
 */
class LightStepTracerFactory(configPath: String, config: Config, loggingProvider: LoggingProvider)
  extends TracerFactory {

  def create(): Tracer = {
    new com.lightstep.tracer.jre.JRETracer(
      new com.lightstep.tracer.shared.Options.OptionsBuilder()
        .withAccessToken("{your_access_token}")
        .build()
    )
  }
}

And then configuring Cinnamon to use this tracer:

cinnamon.opentracing {
  tracers = [lightstep]

  lightstep {
    factory-class = "sample.LightStepTracerFactory"
  }
}