StatsD

StatsD is a commonly used metric collector. StatsD has become more or less a standard in the monitoring domain so we have decided to make it easy to integrate Lightbend Telemetry with StatsD. The following section will describe how to get started with the StatsD plugin.

StatsD does not process events natively. Therefore, we send all event information as metric rates which results in the loss of any contextual information.

Cinnamon dependency

First make sure that your build is configured to use the Cinnamon Agent.

Add the Coda Hale Metrics plugin dependency to your build:

sbt
libraryDependencies += Cinnamon.library.cinnamonCHMetrics
Maven
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-chmetrics_2.11</artifactId>
  <version>2.5.2</version>
</dependency>
Gradle
dependencies {
  compile group: 'com.lightbend.cinnamon', name: 'cinnamon-chmetrics_2.11', version: '2.5.2'
}

Reporter dependency

The next step is to add the StatsD reporter to your build file.

Add the StatsD reporter dependency to your build:

sbt
libraryDependencies += Cinnamon.library.cinnamonCHMetricsStatsDReporter
Maven
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-chmetrics-statsd-reporter_2.11</artifactId>
  <version>2.5.2</version>
</dependency>
Gradle
dependencies {
  compile group: 'com.lightbend.cinnamon', name: 'cinnamon-chmetrics-statsd-reporter_2.11', version: '2.5.2'
}

Configuration

Below is the configuration needed to set up Coda Hale Metrics to use the StatsD reporter. The Reference tab shows all the configurable settings for the StatsD reporter.

Required
cinnamon.chmetrics {
  reporters += "statsd-reporter"
}
Example
cinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    host = "my.statsd.host"
    port = 8125
  }
}
Reference
cinnamon.chmetrics {
  statsd-reporter {

    # Prefix for all metric keys.
    prefix = ""

    # Suffix for all metric keys.
    suffix = ""

    report {
      # Histogram fields to report for metrics
      histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"]

      # Meter fields to report for metrics
      meter = ["samples", "min1_rate", "min5_rate", "min15_rate", "mean_rate"]
    }

    dogstatsd {
      # Whether DogStatsD format should be used
      enabled = off

      # Prefix for all metric keys, when DogStatsD is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Suffix for all metric keys, when DogStatsD is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Tags for all metrics, when DogStatsD is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate tags = "country:UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      # Note: host tag is also already provided by DogStatsD
      unique-dimensions = off
    }

    # Frequency of metric reporting to StatsD.
    frequency = 10s

    # StatsD host.
    host = "127.0.0.1"

    # StatsD port.
    port = 8125
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

For more information about filter settings see Filters.

Note that tags can be used with DogStatsD only.

StatsD wire format information

All metrics will be sent in a StatsD friendly format tailored for usage with the Grafana dashboards that Lightbend provides. Should the need arise, the format can be altered by using prefix and suffix configuration (see above).

Note: All time related metrics are sent as nanoseconds.

Metrics format

servers.<SERVER>.apps.<APPLICATION>.metrics.akka.systems.<ACTOR SYSTEM >.dispatchers.<DISPATCHER>.actors.<ACTOR>.<METRIC TYPE>.<SUB TYPE>:<VALUE>|g

Example values of the above are:

PART VALUE
SERVER 192_168_99_100
APPLICATION cinnamon_sample_bottleneck_BottleneckDemo
ACTOR SYSTEM BottleneckSample
DISPATCHER frontend-dispatcher
ACTOR cinnamon_sample_bottleneck_FrontEndRouter
METRIC TYPE processed-messages
SUB TYPE min1_rate
VALUE 81.11

Note that all package names containing . have been formatted to _, e.g. application value. This is part of the default behavior for the StatsD plugin.

Events format

Events for StatsD have the same layout format as metrics. All events are sent as rates.

Node to node communication format

Akka remoting will involve at least two nodes and any statistics that is related to node to node communication, e.g. Phi accrual value, will have the following format:

<servers>.<SERVER>.apps.<APPLICATION>.metrics.self-nodes.<SELF NODE>.remote-nodes.<REMOTE NODE>.<METRIC TYPE>:<VALUE>|g

Notable above is that there is a self and remote node in the path. The self node can have many remote nodes, but each combination will have its own metric values. The node information is made up from actor system, server and port.

Example values of the above are:

PART VALUE
SERVER 192_168_99_100
APPLICATION cinnamon_sample_bottleneck_BottleneckDemo
SELF NODE MYACTORSYSTEM_192_168_99_100_2551
REMOTE NODE SOMEACTORSYSTEM_192_168_101_255_56010
METRIC TYPE phi-accrual-value
VALUE 0.27

Stopwatch format

Stopwatch format example:

servers.<SERVER>.apps.<APPLICATION>.metrics.stopwatches.<STOPWATCH>.<METRIC TYPE>.<SUB TYPE>:<VALUE>|g

Example values of the above are:

PART VALUE
SERVER 192_168_99_100
APPLICATION cinnamon_sample_bottleneck_BottleneckDemo
STOPWATCH hotpath
METRIC TYPE stopwatch-time
SUB TYPE max
VALUE 42000000

Server and application identifiers

The server and application identifiers reported to StatsD are based on the shared Cinnamon metadata, and will default to the local host name and the Java main class. To configure these, use the cinnamon.host and cinnamon.application settings. For example:

cinnamon {
  host = "somewhere"
  application = "awesome-sauce"
}

DogStatsD

DogStatsD implements the StatsD protocol, along with some extra extensions for Datadog features. The StatsD reporter can emit DogStatsD tags for metric metadata rather than encode this information in the metric key. To enable DogStatsD format, use the dogstatsd.enabled setting.

Required
cinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    dogstatsd {
      enabled = on
  }
}
Example
cinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    host = "my.statsd.host"
    port = 8125

    dogstatsd {
      enabled = on

      # optionally specify tags for all metrics
      tags {
        country = "UK"
      }

      # Optionally include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      unique-dimensions = on
    }
  }
}
Reference
cinnamon.chmetrics {
  statsd-reporter {

    # Prefix for all metric keys.
    prefix = ""

    # Suffix for all metric keys.
    suffix = ""

    report {
      # Histogram fields to report for metrics
      histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"]

      # Meter fields to report for metrics
      meter = ["samples", "min1_rate", "min5_rate", "min15_rate", "mean_rate"]
    }

    dogstatsd {
      # Whether DogStatsD format should be used
      enabled = off

      # Prefix for all metric keys, when DogStatsD is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Suffix for all metric keys, when DogStatsD is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Tags for all metrics, when DogStatsD is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate tags = "country:UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      # Note: host tag is also already provided by DogStatsD
      unique-dimensions = off
    }

    # Frequency of metric reporting to StatsD.
    frequency = 10s

    # StatsD host.
    host = "127.0.0.1"

    # StatsD port.
    port = 8125
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Configuring metric fields

Metrics reported by Cinnamon will be custom metrics in Datadog. As there is a cost associated with the number of custom metrics, it may be useful to limit the metric fields reported for histograms and meters. For example, the following configuration will select particular fields. See the Reference for possible options.

Required

There is nothing to configure if you don’t want to select certain metric fields.

Example
cinnamon.chmetrics {
  reporters += "statsd-reporter"

  statsd-reporter {
    report {
      histogram = ["max", "min", "mean", "p95", "p99"]
      meter = ["samples", "min1_rate"]
    }
  }
}
Reference
cinnamon.chmetrics {
  statsd-reporter {

    # Prefix for all metric keys.
    prefix = ""

    # Suffix for all metric keys.
    suffix = ""

    report {
      # Histogram fields to report for metrics
      histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"]

      # Meter fields to report for metrics
      meter = ["samples", "min1_rate", "min5_rate", "min15_rate", "mean_rate"]
    }

    dogstatsd {
      # Whether DogStatsD format should be used
      enabled = off

      # Prefix for all metric keys, when DogStatsD is enabled
      prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}

      # Suffix for all metric keys, when DogStatsD is enabled
      suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}

      # Tags for all metrics, when DogStatsD is enabled.
      # Uses a "key-value" approach to generate the tags.
      # E.g. the following configuration:
      # tags {
      #   country = "UK"
      # }
      # will generate tags = "country:UK"
      tags {}

      # Whether to include "unique dimensions" as tags.
      # These are tags that are unique to this reporter,
      # such as host name and application identifier.
      # Off by default for compatibility with earlier versions.
      # Note: host tag is also already provided by DogStatsD
      unique-dimensions = off
    }

    # Frequency of metric reporting to StatsD.
    frequency = 10s

    # StatsD host.
    host = "127.0.0.1"

    # StatsD port.
    port = 8125
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

DogStatsD datagram format

When using DogStatsD, the metric key will only represent the type of metric, while information about the Actor, Actor System, or Dispatcher will be provided as tags. As an example, the DogStatsD format for actor metrics is:

actors.<METRIC TYPE>.<SUB TYPE>:<VALUE>|g|#actor-system:<ACTOR SYSTEM>,dispatcher:<DISPATCHER>,actor:<ACTOR>

Example values of the above are:

PART VALUE
METRIC TYPE processed-messages
SUB TYPE min1_rate
VALUE 81.11
ACTOR SYSTEM BottleneckSample
DISPATCHER frontend-dispatcher
ACTOR cinnamon_sample_bottleneck_FrontEndRouter