StatsD is a commonly used metric collector. StatsD has become more or less a standard in the monitoring domain so we have decided to make it easy to integrate Akka Insights with StatsD. The following section will describe how to get started with the StatsD plugin.
StatsD does not process events natively. Therefore, we send all event information as metric rates which results in the loss of any contextual information.
Cinnamon dependency
First make sure that your build is configured to use the Cinnamon Agent.
Reporter dependency
The next step is to add the StatsD reporter to your build file.
Below is the configuration needed to set up Coda Hale Metrics to use the StatsD reporter. The Reference tab shows all the configurable settings for the StatsD reporter.
cinnamon.chmetrics {
statsd-reporter {# StatsD host.
host ="127.0.0.1"# StatsD port.
port =8125# Channel to use for sending statsd messages: udp (by-default) or tcp.
channel = udp
# Frequency of metric reporting to StatsD.
frequency =10s# Prefix for all metric keys.
prefix =""# Prefix for custom metrics. Defaults to `prefix` if not set.
custom-metrics-prefix =null# Suffix for all metric keys.
suffix =""# Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gaugesuse-counters = off
# Whether to enable detailed debug logging for the StatsD reporter# This also requires enabling DEBUG in your logger for class:# `com.lightbend.cinnamon.chmetrics.statsd.DebugStatsDClient`# AND setting the `akka.loglevel` configuration to DEBUG# Note: only use for troubleshooting as every packet send is logged
debug = off
# If debug logging is enabled, only debug every n:th report
debug-every =1
report {# When `use-counters` is enabled, also report the total count as an additional `.total` gauge
counter-totals = off
# Histogram fields to report for metrics
histogram =["max","min","mean","median","stddev","p75","p95","p98","p99","p999"]# Meter fields to report for metrics
meter =["samples","samples.total","min1_rate","min5_rate","min15_rate","mean_rate"]}
dogstatsd {# Whether DogStatsD format should be used
enabled = off
# Prefix for all metric keys, when DogStatsD is enabled
prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}# Prefix for custom metrics, when DogStatsD is enabled. Defaults to `prefix` if not set.
custom-metrics-prefix =null# Suffix for all metric keys, when DogStatsD is enabled
suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}# Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gaugesuse-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}# Tags for all metrics, when DogStatsD is enabled.# Uses a "key-value" approach to generate the tags.# E.g. the following configuration:# tags {# country = "UK"# }# will generate this Datadog tag: "country:UK"
tags {}# Whether to include "unique dimensions" as tags.# These are tags that are unique to this reporter,# such as host name and application identifier.# Off by default for compatibility with earlier versions.# Note: host tag is also already provided by DogStatsD
unique-dimensions = off
}
telegraf {# Whether Telegraf format should be used or not
enabled = off
# Prefix for all metric keys, when Telegraf is enabled
prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}# Prefix for custom metrics, when Telegraf is enabled. Defaults to `prefix` if not set.
custom-metrics-prefix =null# Suffix for all metric keys, when Telegraf is enabled
suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}# Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gaugesuse-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}# Tags for all metrics, when Telegraf is enabled.# Uses a "key-value" approach to generate the tags.# E.g. the following configuration:# tags {# country = "UK"# }# will generate this Telegraf tag: "country=UK"
tags {}# Whether to include "unique dimensions" as tags.# These are tags that are unique to this reporter,# such as host name and application identifier.# Off by default for compatibility with earlier versions.
unique-dimensions = off
}}}
@@@ note These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.
For more information about filter settings see Filters.
Note that tags can be used with DogStatsD only. @@@
Note
By default StatsD is configured to use the UDP reporter that may not work correctly with Docker on MacOS. Use the TCP reporter instead as shown in the example below.
All metrics will be sent in a StatsD friendly format tailored for usage with the Grafana dashboards that Lightbend provides. Should the need arise, the format can be altered by using prefix and suffix configuration (see above).
Note
All time related metrics are sent as nanoseconds.
Metrics format
servers.<SERVER>.apps.<APPLICATION>.metrics.akka.systems.<ACTOR SYSTEM >.dispatchers.<DISPATCHER>.actors.<ACTOR>.<METRIC TYPE>.<SUB TYPE>:<VALUE>|g
Example values of the above are:
PART
VALUE
SERVER
192_168_99_100
APPLICATION
cinnamon_sample_bottleneck_BottleneckDemo
ACTOR SYSTEM
BottleneckSample
DISPATCHER
frontend-dispatcher
ACTOR
cinnamon_sample_bottleneck_FrontEndRouter
METRIC TYPE
processed-messages
SUB TYPE
min1_rate
VALUE
81.11
Note that all package names containing . have been formatted to _, e.g. application value. This is part of the default behavior for the StatsD plugin.
Events format
Events for StatsD have the same layout format as metrics. All events are sent as rates.
Node to node communication format
Akka remoting will involve at least two nodes and any statistics that is related to node to node communication, e.g. Phi accrual value, will have the following format:
Notable above is that there is a self and remote node in the path. The self node can have many remote nodes, but each combination will have its own metric values. The node information is made up from actor system, server and port.
The server and application identifiers reported to StatsD are based on the shared Cinnamon metadata, and will default to the local host name and the Java main class. To configure these, use the cinnamon.host and cinnamon.application settings. For example:
By default all metrics are reported as StatsD gauges. It is possible to have the Coda Hale counters and the samples field of Coda Hale meters reported as the delta between the current value and the last reported valus as a StatsD counter, by changing the configuration like this:
cinnamon.chmetrics {
reporters +="statsd-reporter"
statsd-reporter {# use delta-based countersuse-counters = on
}}
It’s also possible to have both the delta-based counter and the total count reported as a gauge. The total count will be an additional gauge metric, named with a .total suffix. Enable the counter-totals setting in the StatsD reporting configuration:
cinnamon.chmetrics {
reporters +="statsd-reporter"
statsd-reporter {# use delta-based countersuse-counters = on
report {# report both delta-based counters and totals
counter-totals = on
}}}
DogStatsD
DogStatsD implements the StatsD protocol, along with some extra extensions for Datadog features. The StatsD reporter can emit DogStatsD tags for metric metadata rather than encode this information in the metric key. To enable DogStatsD format, use the dogstatsd.enabled setting.
cinnamon.chmetrics {
reporters +="statsd-reporter"
statsd-reporter {
host ="my.statsd.host"
port =8125
dogstatsd {
enabled = on
# optionally specify tags for all metrics
tags {
country ="UK"}# Optionally include "unique dimensions" as tags.# These are tags that are unique to this reporter,# such as host name and application identifier.
unique-dimensions = on
}}}
cinnamon.chmetrics {
statsd-reporter {# StatsD host.
host ="127.0.0.1"# StatsD port.
port =8125# Channel to use for sending statsd messages: udp (by-default) or tcp.
channel = udp
# Frequency of metric reporting to StatsD.
frequency =10s# Prefix for all metric keys.
prefix =""# Prefix for custom metrics. Defaults to `prefix` if not set.
custom-metrics-prefix =null# Suffix for all metric keys.
suffix =""# Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gaugesuse-counters = off
# Whether to enable detailed debug logging for the StatsD reporter# This also requires enabling DEBUG in your logger for class:# `com.lightbend.cinnamon.chmetrics.statsd.DebugStatsDClient`# AND setting the `akka.loglevel` configuration to DEBUG# Note: only use for troubleshooting as every packet send is logged
debug = off
# If debug logging is enabled, only debug every n:th report
debug-every =1
report {# When `use-counters` is enabled, also report the total count as an additional `.total` gauge
counter-totals = off
# Histogram fields to report for metrics
histogram =["max","min","mean","median","stddev","p75","p95","p98","p99","p999"]# Meter fields to report for metrics
meter =["samples","samples.total","min1_rate","min5_rate","min15_rate","mean_rate"]}
dogstatsd {# Whether DogStatsD format should be used
enabled = off
# Prefix for all metric keys, when DogStatsD is enabled
prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}# Prefix for custom metrics, when DogStatsD is enabled. Defaults to `prefix` if not set.
custom-metrics-prefix =null# Suffix for all metric keys, when DogStatsD is enabled
suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}# Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gaugesuse-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}# Tags for all metrics, when DogStatsD is enabled.# Uses a "key-value" approach to generate the tags.# E.g. the following configuration:# tags {# country = "UK"# }# will generate this Datadog tag: "country:UK"
tags {}# Whether to include "unique dimensions" as tags.# These are tags that are unique to this reporter,# such as host name and application identifier.# Off by default for compatibility with earlier versions.# Note: host tag is also already provided by DogStatsD
unique-dimensions = off
}
telegraf {# Whether Telegraf format should be used or not
enabled = off
# Prefix for all metric keys, when Telegraf is enabled
prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}# Prefix for custom metrics, when Telegraf is enabled. Defaults to `prefix` if not set.
custom-metrics-prefix =null# Suffix for all metric keys, when Telegraf is enabled
suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}# Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gaugesuse-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}# Tags for all metrics, when Telegraf is enabled.# Uses a "key-value" approach to generate the tags.# E.g. the following configuration:# tags {# country = "UK"# }# will generate this Telegraf tag: "country=UK"
tags {}# Whether to include "unique dimensions" as tags.# These are tags that are unique to this reporter,# such as host name and application identifier.# Off by default for compatibility with earlier versions.
unique-dimensions = off
}}}
Note
These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.
Also see the Datadog plugin for default DogStatsD configuration for reporting to Datadog.
Configuring metric fields
Metrics reported by Cinnamon will be custom metrics in Datadog. As there is a cost associated with the number of custom metrics, it may be useful to limit the metric fields reported for histograms and meters. For example, the following configuration will select particular fields. See the Reference for possible options.
cinnamon.chmetrics {
statsd-reporter {# StatsD host.
host ="127.0.0.1"# StatsD port.
port =8125# Channel to use for sending statsd messages: udp (by-default) or tcp.
channel = udp
# Frequency of metric reporting to StatsD.
frequency =10s# Prefix for all metric keys.
prefix =""# Prefix for custom metrics. Defaults to `prefix` if not set.
custom-metrics-prefix =null# Suffix for all metric keys.
suffix =""# Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gaugesuse-counters = off
# Whether to enable detailed debug logging for the StatsD reporter# This also requires enabling DEBUG in your logger for class:# `com.lightbend.cinnamon.chmetrics.statsd.DebugStatsDClient`# AND setting the `akka.loglevel` configuration to DEBUG# Note: only use for troubleshooting as every packet send is logged
debug = off
# If debug logging is enabled, only debug every n:th report
debug-every =1
report {# When `use-counters` is enabled, also report the total count as an additional `.total` gauge
counter-totals = off
# Histogram fields to report for metrics
histogram =["max","min","mean","median","stddev","p75","p95","p98","p99","p999"]# Meter fields to report for metrics
meter =["samples","samples.total","min1_rate","min5_rate","min15_rate","mean_rate"]}
dogstatsd {# Whether DogStatsD format should be used
enabled = off
# Prefix for all metric keys, when DogStatsD is enabled
prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}# Prefix for custom metrics, when DogStatsD is enabled. Defaults to `prefix` if not set.
custom-metrics-prefix =null# Suffix for all metric keys, when DogStatsD is enabled
suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}# Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gaugesuse-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}# Tags for all metrics, when DogStatsD is enabled.# Uses a "key-value" approach to generate the tags.# E.g. the following configuration:# tags {# country = "UK"# }# will generate this Datadog tag: "country:UK"
tags {}# Whether to include "unique dimensions" as tags.# These are tags that are unique to this reporter,# such as host name and application identifier.# Off by default for compatibility with earlier versions.# Note: host tag is also already provided by DogStatsD
unique-dimensions = off
}
telegraf {# Whether Telegraf format should be used or not
enabled = off
# Prefix for all metric keys, when Telegraf is enabled
prefix = ${cinnamon.chmetrics.statsd-reporter.prefix}# Prefix for custom metrics, when Telegraf is enabled. Defaults to `prefix` if not set.
custom-metrics-prefix =null# Suffix for all metric keys, when Telegraf is enabled
suffix = ${cinnamon.chmetrics.statsd-reporter.suffix}# Send Counter `values`, and Meter `samples` as StatsD Counters instead of Gaugesuse-counters = ${cinnamon.chmetrics.statsd-reporter.use-counters}# Tags for all metrics, when Telegraf is enabled.# Uses a "key-value" approach to generate the tags.# E.g. the following configuration:# tags {# country = "UK"# }# will generate this Telegraf tag: "country=UK"
tags {}# Whether to include "unique dimensions" as tags.# These are tags that are unique to this reporter,# such as host name and application identifier.# Off by default for compatibility with earlier versions.
unique-dimensions = off
}}}
Note
These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.
DogStatsD datagram format
When using DogStatsD, the metric key will only represent the type of metric, while information about the Actor, Actor System, or Dispatcher will be provided as tags. As an example, the DogStatsD format for actor metrics is: