Datadog

We have partnered with Datadog to provide a plugin for Lightbend Telemetry for Akka with Datadog.

Cinnamon dependency

We provide an easy-to-get-started plugin that contains all settings required for pushing Lightbend related metrics to Datadog.

First make sure that your build is configured to use the Cinnamon Agent.

Add the core Coda Hale Metrics dependency and the Cinnamon Datadog dependency to your build:

sbt
libraryDependencies += Cinnamon.library.cinnamonCHMetrics
libraryDependencies += Cinnamon.library.cinnamonDatadog
Maven
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-chmetrics</artifactId>
  <version>2.11.1</version>
</dependency>
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-datadog</artifactId>
  <version>2.11.1</version>
</dependency>
Gradle
dependencies {
  compile group: 'com.lightbend.cinnamon', name: 'cinnamon-chmetrics', version: '2.11.1'
}
dependencies {
  compile group: 'com.lightbend.cinnamon', name: 'cinnamon-datadog', version: '2.11.1'
}

Note: The default Coda Hale Metrics module uses Metrics 4 but Coda Hale Metrics 3 can also be used.

The Cinnamon Datadog reporter provides a configured StatsD reporter. The default values can be overridden should you want to change the default behavior.

Required
cinnamon.chmetrics {
  reporters += datadog-reporter
}
Example
cinnamon.chmetrics {
  reporters += datadog-reporter

  datadog-reporter {
    host = "192.168.0.1"
    port = 8888
    frequency = 60s

    report {
      histogram = ["min", "max", "p98", "p99", "p999"]
    }
  }
}
Datadog Reference
cinnamon.chmetrics {
  datadog-reporter = ${cinnamon.chmetrics.statsd-reporter} {
    dogstatsd {
      enabled = on

      # Prefix must be "lightbend" to enable the Lightbend Reactive Platform integration in Datadog
      prefix = "lightbend"
    }

    report {
      # Histogram fields to report for metrics
      histogram = ["max", "p99"]

      # Meter fields to report for metrics
      meter = ["min1_rate"]
    }
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

StatsD Reference
cinnamon.chmetrics {
  statsd-reporter {

    # StatsD host.
    host = "127.0.0.1"

    # StatsD port.
    port = 8125

    # Channel to use for sending statsd messages.
    channel = udp

    # Frequency of metric reporting to StatsD.
    frequency = 10s

    # Prefix for all metric keys.
    prefix = ""

    # Suffix for all metric keys.
    suffix = ""

    # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
    use-counters = off

    # Whether to enable detailed debug logging for the StatsD reporter
    # This also requires enabling DEBUG in your logger for class:
    #   `com.lightbend.cinnamon.chmetrics.statsd.DebugStatsDClient`
    # AND setting the `akka.loglevel` configuration to DEBUG
    # Note: only use for troubleshooting as every packet send is logged
    debug = off

    # If debug logging is enabled, only debug every n:th report
    debug-every = 1

    report {
      # Histogram fields to report for metrics
      histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"]

      # Meter fields to report for metrics
      meter = ["samples", "min1_rate", "min5_rate", "min15_rate", "mean_rate"]
    }

    dogstatsd {
      # Whether DogStatsD format should be used
      enabled = off

      # Prefix for all metric keys, when DogStatsD is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Suffix for all metric keys, when DogStatsD is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
      use-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}

      # Tags for all metrics, when DogStatsD is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate this Datadog tag: "country:UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      # Note: host tag is also already provided by DogStatsD
      unique-dimensions = off
    }

    telegraf {
      # Whether Telegraf format should be used or not
      enabled = off

      # Prefix for all metric keys, when Telegraf is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Suffix for all metric keys, when Telegraf is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
      use-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}

      # Tags for all metrics, when Telegraf is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate this Telegraf tag: "country=UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      unique-dimensions = off
    }
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Reporting deltas as counters

By default all metrics are reported as StatsD gauges. It is possible to have the Coda Hale counters and the samples field of Coda Hale meters reported as the delta between the current value and the last reported valus as a StatsD counter, by changing the configuration like this:

cinnamon.chmetrics {
  reporters += "datadog-reporter"

  datadog-reporter.dogstatsd {
    use-counters = on
  }
}

Unix Domain Socket

The Datadog Agent is able to ingest metrics via a Unix Domain Socket, as an alternative to UDP, when running on Linux systems. The Cinnamon Datadog reporter also supports sending metrics via a Unix Domain Socket.

To enable Unix Domain Socket support, use the cinnamon-datadog-socket dependency instead of the regular cinnamon-datadog dependency:

sbt
libraryDependencies += Cinnamon.library.cinnamonDatadogSocket
Maven
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-datadog-socket</artifactId>
  <version>2.11.1</version>
</dependency>
Gradle
dependencies {
  compile group: 'com.lightbend.cinnamon', name: 'cinnamon-datadog-socket', version: '2.11.1'
}

Then enable Unix Domain Socket support using configuration, providing the path to the socket:

Example
cinnamon.chmetrics {
  reporters += "datadog-reporter"

  datadog-reporter {
    channel = unix-domain-socket
    unix-domain-socket {
      path = /var/run/datadog/dsd.socket
    }
  }
}
Reference
cinnamon.chmetrics {
  statsd-reporter {
    unix-domain-socket {
      # If configured then statsd reporting will be via unix domain datagram socket at this path.
      path = null

      # SO_SNDBUF setting (in config bytes) for the unix domain datagram socket, otherwise system default.
      send-buffer-size = null

      # SO_SNDTIMEO setting (in config duration) for the unix domain datagram socket, otherwise system default.
      send-timeout = null
    }
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Datadog dependency

For information about how to get started with Datadog we refer to their excellent getting started instructions. Go to their website, create an account and follow the instructions for your particular environment.