StatsD

StatsD is a commonly used metric collector. StatsD has become more or less a standard in the monitoring domain so we have decided to make it easy to integrate Lightbend Telemetry with StatsD. The following section will describe how to get started with the StatsD plugin.

StatsD does not process events natively. Therefore, we send all event information as metric rates which results in the loss of any contextual information.

Cinnamon dependency

First make sure that your build is configured to use the Cinnamon Agent.

Reporter dependency

The next step is to add the StatsD reporter to your build file.

Add the core Coda Hale Metrics dependency and the StatsD reporter dependency to your build:

sbt

copylibraryDependencies += Cinnamon.library.cinnamonCHMetrics
libraryDependencies += Cinnamon.library.cinnamonCHMetricsStatsDReporter

Maven

copy<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-chmetrics</artifactId>
    <version>2.19.3</version>
</dependency>
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-chmetrics-statsd-reporter</artifactId>
  <version>2.19.3</version>
</dependency>

Gradle

copydependencies {
  implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-chmetrics', version: '2.19.3'
}
dependencies {
  implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-chmetrics-statsd-reporter', version: '2.19.3'
}

Configuration

Below is the configuration needed to set up Coda Hale Metrics to use the StatsD reporter. The Reference tab shows all the configurable settings for the StatsD reporter.

Required

copycinnamon.chmetrics {
  reporters += "statsd-reporter"
}

Example

copycinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    host = "my.statsd.host"
    port = 8125
  }
}

Reference

copycinnamon.chmetrics {
  statsd-reporter {

    # StatsD host.
    host = "127.0.0.1"

    # StatsD port.
    port = 8125

    # Channel to use for sending statsd messages: udp (by-default) or tcp.
    channel = udp

    # Frequency of metric reporting to StatsD.
    frequency = 10s

    # Prefix for all metric keys.
    prefix = ""

    # Prefix for custom metrics. Defaults to `prefix` if not set.
    custom-metrics-prefix = null

    # Suffix for all metric keys.
    suffix = ""

    # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
    use-counters = off

    # Whether to enable detailed debug logging for the StatsD reporter
    # This also requires enabling DEBUG in your logger for class:
    #   `com.lightbend.cinnamon.chmetrics.statsd.DebugStatsDClient`
    # AND setting the `akka.loglevel` configuration to DEBUG
    # Note: only use for troubleshooting as every packet send is logged
    debug = off

    # If debug logging is enabled, only debug every n:th report
    debug-every = 1

    report {
      # When `use-counters` is enabled, also report the total count as an additional `.total` gauge
      counter-totals = off

      # Histogram fields to report for metrics
      histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"]

      # Meter fields to report for metrics
      meter = ["samples", "samples.total", "min1_rate", "min5_rate", "min15_rate", "mean_rate"]
    }

    dogstatsd {
      # Whether DogStatsD format should be used
      enabled = off

      # Prefix for all metric keys, when DogStatsD is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Prefix for custom metrics, when DogStatsD is enabled. Defaults to `prefix` if not set.
      custom-metrics-prefix = null

      # Suffix for all metric keys, when DogStatsD is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
      use-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}

      # Tags for all metrics, when DogStatsD is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate this Datadog tag: "country:UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      # Note: host tag is also already provided by DogStatsD
      unique-dimensions = off
    }

    telegraf {
      # Whether Telegraf format should be used or not
      enabled = off

      # Prefix for all metric keys, when Telegraf is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Prefix for custom metrics, when Telegraf is enabled. Defaults to `prefix` if not set.
      custom-metrics-prefix = null

      # Suffix for all metric keys, when Telegraf is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
      use-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}

      # Tags for all metrics, when Telegraf is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate this Telegraf tag: "country=UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      unique-dimensions = off
    }
  }
}

@@@ note These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

For more information about filter settings see Filters.

Note that tags can be used with DogStatsD only. @@@

Example

copycinnamon.chmetrics {
  reporters += "statsd-reporter"
  statsd-reporter {
    channel = tcp
  }
}

StatsD wire format information

All metrics will be sent in a StatsD friendly format tailored for usage with the Grafana dashboards that Lightbend provides. Should the need arise, the format can be altered by using prefix and suffix configuration (see above).

Metrics format

servers.<SERVER>.apps.<APPLICATION>.metrics.akka.systems.<ACTOR SYSTEM >.dispatchers.<DISPATCHER>.actors.<ACTOR>.<METRIC TYPE>.<SUB TYPE>:<VALUE>|g

Example values of the above are:

Note that all package names containing . have been formatted to _, e.g. application value. This is part of the default behavior for the StatsD plugin.

Events format

Events for StatsD have the same layout format as metrics. All events are sent as rates.

Node to node communication format

Akka remoting will involve at least two nodes and any statistics that is related to node to node communication, e.g. Phi accrual value, will have the following format:

<servers>.<SERVER>.apps.<APPLICATION>.metrics.self-nodes.<SELF NODE>.remote-nodes.<REMOTE NODE>.<METRIC TYPE>:<VALUE>|g

Notable above is that there is a self and remote node in the path. The self node can have many remote nodes, but each combination will have its own metric values. The node information is made up from actor system, server and port.

Example values of the above are:

Stopwatch format

Stopwatch format example:

servers.<SERVER>.apps.<APPLICATION>.metrics.stopwatches.<STOPWATCH>.<METRIC TYPE>.<SUB TYPE>:<VALUE>|g

Example values of the above are:

Server and application identifiers

The server and application identifiers reported to StatsD are based on the shared Cinnamon metadata, and will default to the local host name and the Java main class. To configure these, use the cinnamon.host and cinnamon.application settings. For example:

copycinnamon {
  host = "somewhere"
  application = "awesome-sauce"
}

Reporting deltas as counters

By default all metrics are reported as StatsD gauges. It is possible to have the Coda Hale counters and the samples field of Coda Hale meters reported as the delta between the current value and the last reported valus as a StatsD counter, by changing the configuration like this:

copycinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    # use delta-based counters
    use-counters = on
  }
}

It’s also possible to have both the delta-based counter and the total count reported as a gauge. The total count will be an additional gauge metric, named with a .total suffix. Enable the counter-totals setting in the StatsD reporting configuration:

copycinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    # use delta-based counters
    use-counters = on

    report {
      # report both delta-based counters and totals
      counter-totals = on
    }
  }
}

DogStatsD

DogStatsD implements the StatsD protocol, along with some extra extensions for Datadog features. The StatsD reporter can emit DogStatsD tags for metric metadata rather than encode this information in the metric key. To enable DogStatsD format, use the dogstatsd.enabled setting.

Required

copycinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    dogstatsd {
      enabled = on
    }
  }
}

Example

copycinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    host = "my.statsd.host"
    port = 8125

    dogstatsd {
      enabled = on

      # optionally specify tags for all metrics
      tags {
        country = "UK"
      }

      # Optionally include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      unique-dimensions = on
    }
  }
}

Reference

copycinnamon.chmetrics {
  statsd-reporter {

    # StatsD host.
    host = "127.0.0.1"

    # StatsD port.
    port = 8125

    # Channel to use for sending statsd messages: udp (by-default) or tcp.
    channel = udp

    # Frequency of metric reporting to StatsD.
    frequency = 10s

    # Prefix for all metric keys.
    prefix = ""

    # Prefix for custom metrics. Defaults to `prefix` if not set.
    custom-metrics-prefix = null

    # Suffix for all metric keys.
    suffix = ""

    # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
    use-counters = off

    # Whether to enable detailed debug logging for the StatsD reporter
    # This also requires enabling DEBUG in your logger for class:
    #   `com.lightbend.cinnamon.chmetrics.statsd.DebugStatsDClient`
    # AND setting the `akka.loglevel` configuration to DEBUG
    # Note: only use for troubleshooting as every packet send is logged
    debug = off

    # If debug logging is enabled, only debug every n:th report
    debug-every = 1

    report {
      # When `use-counters` is enabled, also report the total count as an additional `.total` gauge
      counter-totals = off

      # Histogram fields to report for metrics
      histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"]

      # Meter fields to report for metrics
      meter = ["samples", "samples.total", "min1_rate", "min5_rate", "min15_rate", "mean_rate"]
    }

    dogstatsd {
      # Whether DogStatsD format should be used
      enabled = off

      # Prefix for all metric keys, when DogStatsD is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Prefix for custom metrics, when DogStatsD is enabled. Defaults to `prefix` if not set.
      custom-metrics-prefix = null

      # Suffix for all metric keys, when DogStatsD is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
      use-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}

      # Tags for all metrics, when DogStatsD is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate this Datadog tag: "country:UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      # Note: host tag is also already provided by DogStatsD
      unique-dimensions = off
    }

    telegraf {
      # Whether Telegraf format should be used or not
      enabled = off

      # Prefix for all metric keys, when Telegraf is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Prefix for custom metrics, when Telegraf is enabled. Defaults to `prefix` if not set.
      custom-metrics-prefix = null

      # Suffix for all metric keys, when Telegraf is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
      use-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}

      # Tags for all metrics, when Telegraf is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate this Telegraf tag: "country=UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      unique-dimensions = off
    }
  }
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Also see the Datadog plugin for default DogStatsD configuration for reporting to Datadog.

Configuring metric fields

Metrics reported by Cinnamon will be custom metrics in Datadog. As there is a cost associated with the number of custom metrics, it may be useful to limit the metric fields reported for histograms and meters. For example, the following configuration will select particular fields. See the Reference for possible options.

Required

There is nothing to configure if you don’t want to select certain metric fields.

Example

copycinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    report {
      histogram = ["max", "min", "mean", "p95", "p99"]
      meter = ["samples", "min1_rate"]
    }
  }
}

Reference

copycinnamon.chmetrics {
  statsd-reporter {

    # StatsD host.
    host = "127.0.0.1"

    # StatsD port.
    port = 8125

    # Channel to use for sending statsd messages: udp (by-default) or tcp.
    channel = udp

    # Frequency of metric reporting to StatsD.
    frequency = 10s

    # Prefix for all metric keys.
    prefix = ""

    # Prefix for custom metrics. Defaults to `prefix` if not set.
    custom-metrics-prefix = null

    # Suffix for all metric keys.
    suffix = ""

    # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
    use-counters = off

    # Whether to enable detailed debug logging for the StatsD reporter
    # This also requires enabling DEBUG in your logger for class:
    #   `com.lightbend.cinnamon.chmetrics.statsd.DebugStatsDClient`
    # AND setting the `akka.loglevel` configuration to DEBUG
    # Note: only use for troubleshooting as every packet send is logged
    debug = off

    # If debug logging is enabled, only debug every n:th report
    debug-every = 1

    report {
      # When `use-counters` is enabled, also report the total count as an additional `.total` gauge
      counter-totals = off

      # Histogram fields to report for metrics
      histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"]

      # Meter fields to report for metrics
      meter = ["samples", "samples.total", "min1_rate", "min5_rate", "min15_rate", "mean_rate"]
    }

    dogstatsd {
      # Whether DogStatsD format should be used
      enabled = off

      # Prefix for all metric keys, when DogStatsD is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Prefix for custom metrics, when DogStatsD is enabled. Defaults to `prefix` if not set.
      custom-metrics-prefix = null

      # Suffix for all metric keys, when DogStatsD is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
      use-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}

      # Tags for all metrics, when DogStatsD is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate this Datadog tag: "country:UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      # Note: host tag is also already provided by DogStatsD
      unique-dimensions = off
    }

    telegraf {
      # Whether Telegraf format should be used or not
      enabled = off

      # Prefix for all metric keys, when Telegraf is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Prefix for custom metrics, when Telegraf is enabled. Defaults to `prefix` if not set.
      custom-metrics-prefix = null

      # Suffix for all metric keys, when Telegraf is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gauges
      use-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}

      # Tags for all metrics, when Telegraf is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate this Telegraf tag: "country=UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      unique-dimensions = off
    }
  }
}

Note

These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

DogStatsD datagram format

When using DogStatsD, the metric key will only represent the type of metric, while information about the Actor, Actor System, or Dispatcher will be provided as tags. As an example, the DogStatsD format for actor metrics is:

actors.<METRIC TYPE>.<SUB TYPE>:<VALUE>|g|#actor-system:<ACTOR SYSTEM>,dispatcher:<DISPATCHER>,actor:<ACTOR>

Example values of the above are: