New Relic

Cinnamon provides a plugin that integrates with New Relic Insights using the New Relic Agent. This makes all Cinnamon metrics available as structured New Relic Insights events with rich metadata tags.

Cinnamon dependency

We provide an easy-to-get-started plugin that contains all settings required for pushing Lightbend related metrics to New Relic.

This plugin is called cinnamonNewRelic and you add it like this:

sbt
Cinnamon.library.cinnamonNewRelic
Maven
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-newrelic_2.12</artifactId>
  <version>2.10.3</version>
</dependency>
Gradle
compile group: 'com.lightbend.cinnamon', name: 'cinnamon-newrelic_2.12', version: '2.10.3'

The Cinnamon New Relic reporter provides a preconfigured reporter for our Coda Hale Metrics integration. Here is how you enable the reporter, and an example of how to override some default values if you want to change the default behavior.

Required
cinnamon.chmetrics {
  reporters += newrelic-reporter
}
Example
cinnamon.chmetrics {
  reporters += newrelic-reporter

  newrelic-reporter {
    frequency = 60s

    report {
      histogram = ["min", "max", "p98", "p99", "p999"]
    }
  }
}
Reference
cinnamon.chmetrics {
  newrelic-reporter {

    # Frequency of metric reporting to NewRelic Agent.
    frequency = 10s

    # Prefix for all metric keys.
    prefix = "lightbend"

    # Suffix for all metric keys.
    suffix = ""

    # Tags for all metrics.
    # Uses a "key-value" approach to generate the tags.
    # E.g. the following configuration:
    # tags {
    #   country = "UK"
    # }
    # will generate tags = "country" -> "UK"
    tags {}

    # Handle events.
    # Control if the reporter should send events to new relic as well.
    handle-events = false

    report {
      # Histogram fields to report for metrics.
      histogram = ["max", "min", "mean", "median", "stddev", "p75", "p95", "p98", "p99", "p999"]

      # Meter fields to report for metrics.
      meter = ["samples", "min1_rate", "min5_rate", "min15_rate", "mean_rate"]

      # Tags to include for metrics. Does not affect histogram or meter fields.
      include-tags = ["*"]

      # Tags to exclude for metrics. Does not affect histogram or meter fields.
      exclude-tags = []

      # Converts time from nano into specified unit. Use "ns" for no conversion.
      # Supported units: "ns" (default), "us", "ms", and "s".
      conversion-time-unit = "ns"

      # Group by category instead of name.
      # If enabled the New Relic event names will only include the metric category and not the name,
      # and a new tag named "metric" containing the metric name will be added.
      # E.g. the metric "lightbend:actors:mailbox_size" will be reported as "lightbend:actors" with
      # the tag "metric" -> "mailbox_size"
      group-by-category = false
    }
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

Event reporting

By default all Cinnamon metrics are reported to New Relic, but it is also possible to report Cinnamon events to New Relic by changing the configuration like this:

cinnamon.chmetrics.newrelic-reporter.handle-events = true

To manage the amount of data being reported to New Relic, you can disable or rate limit events via configuration.

Note: The total number of times that an event has fired is automatically reported as a metric, so there will be one instance with type = metric and one instance with type = event with the same name.

Format

The information published to New Relic as Custom Events have their names formatted in this fashion:

[<prefix>:]<category>:<name>[:<suffix>]

As an example, the name of the akka actor mailbox size metric with the default configuration would be:

lighbend:actors:mailbox_size

All metrics and events have a field named type that is set to metric or event respectively.

Grouping by category

There is also the option to have the metrics and events, that are being sent to New Relic, grouped together at the category level, to make the number of names showing up in the New Relic UI smaller. To group the names together you need to change the configuration like this.

cinnamon.chmetrics.newrelic-reporter.report.group-by-category = true

All metrics and events will now have a field named name, that contains the metric or event name, and the name of Custom Event reported to New Relic will be.

[<prefix>:]<category>[:<suffix>]

The same example for akka actor mailbox size will now have a Custom Event with the name lighbend:actors, as well as a field name containing mailbox_size.

New Relic time units

New Relic has support for time data formatting of Insight Events tags, but only for milliseconds and seconds. By default Cinnamon will report all time related metrics to New Relic in nanoseconds. See settings in the reference.conf for more information of how to configure the behavior.

New Relic queries

To plot the data that is reported into New Relic Insights, you need to create your own graphs with some queries.

This is a sample query for plotting the akka actor mailbox size metric per actor:

SELECT average(mean) FROM `lightbend:actors:mailbox_size` FACET actor WHERE type = 'metric' SINCE 30 MINUTES AGO TIMESERIES

And the same query when metrics and events are grouped by category:

SELECT average(mean) FROM `lightbend:actors` FACET actor WHERE type = 'metric' AND name = 'mailbox_size' SINCE 30 MINUTES AGO TIMESERIES

New Relic dependency

For information about how to get started with New Relic we refer to their getting started instructions. Go to their website, create an account and follow the instructions for your particular environment. Please note that the Cinnamon New Relic integration uses the New Relic Agent to send information to New Relic Insights, so that agent needs to be enabled and configured.

Note: The New Relic Agent is known to interfere with some of the instrumentations that the Cinnamon Agent does, to Akka, Play and Scala, So these must be disabled in the New Relic Agent.

There are instructions on the New Relic site, on how to disable specific async frameworks and modules. Here is the complete list of modules to disable as of version 3.48.0 of the New Relic Agent.

  class_transformer:
    # Disable Akka, Akka HTTP, Play, and Scala instrumentation
    com.newrelic.instrumentation.akka-2.0:
      enabled: false
    com.newrelic.instrumentation.akka-2.1:
      enabled: false
    com.newrelic.instrumentation.akka-2.2:
      enabled: false
    com.newrelic.instrumentation.akka-http-core-10.0:
      enabled: false
    com.newrelic.instrumentation.akka-http-core-1.0:
      enabled: false
    com.newrelic.instrumentation.akka-http-core-0.7:
      enabled: false
    com.newrelic.instrumentation.akka-http-core-0.4:
      enabled: false
    com.newrelic.instrumentation.akka-http-2.4.5:
      enabled: false
    com.newrelic.instrumentation.akka-http-2.4.2:
      enabled: false
    com.newrelic.instrumentation.akka-http-2.0:
      enabled: false
    com.newrelic.instrumentation.akka-http-1.0:
      enabled: false
    com.newrelic.instrumentation.play-2.6:
      enabled: false
    com.newrelic.instrumentation.play-2.5:
      enabled: false
    com.newrelic.instrumentation.play-2.4:
      enabled: false
    com.newrelic.instrumentation.play-2.3:
      enabled: false
    com.newrelic.instrumentation.play-2.2:
      enabled: false
    com.newrelic.instrumentation.play-2.1:
      enabled: false
    com.newrelic.instrumentation.play-2.0:
      enabled: false
    com.newrelic.instrumentation.play-ws-2.6.0:
      enabled: false
    com.newrelic.instrumentation.scala-2.12.0:
      enabled: false
    com.newrelic.instrumentation.scala-2.9.3:
      enabled: false