Grafana

Lightbend Telemetry provides dashboard templates that one can import into Grafana. This section describes how to install them into Grafana and what information they provide.

Prerequisite software

Before you can start using the Grafana templates, you must install a foundation for getting data from Lightbend Telemetry into Grafana. Although there are multiple ways of setting this up this guide will use a Grafana/Graphite/StatsD combination.

Follow the installation instructions for each of the tools above and make sure that they are working before continuing.

Cinnamon Dashboards

Lightbend Telemetry provides several predefined dashboard templates including:

  • Actor metrics : miscellaneous metrics for actors
  • Events : actor events, e.g. dead letters, unhandled messages
  • Stopwatch metrics: information about Stopwatches that have been programmatically added
  • JVM information : heap and garbage collection information
  • Lagom circuit breakers : circuit breaker status and metrics

Here below you can see example dashboards for Actor and JVM information:

Actor dashboard

Actor dashboard

Installation

Make sure to install the tools in the prerequisite’s section here above. Run Grafana and ensure that you can access the UI on http://localhost.

The next step is to login on http://localhost/login. The default admin and password combination is admin and admin. Once logged in as administrator on Grafana you can proceed with the steps below.

Add Graphite datasource

Make sure that there is a datasource connection that can be used to retrieve data from.

  1. Click Datasources -> Add New
  2. Give the DS a name, e.g. CinnamonGraphiteDS
  3. Check the default flag
  4. Type should be Graphite
  5. Set URL to http://localhost:8000
  6. Click Add
  7. Verify DS by clicking Test Connection
  8. Click Save

Import dashboards

Follow this set of steps to import a dashboard. The predefined dashboards can be found in the Cinnamon UI repository in the Grafana section.

  1. In the top menu, click on Home -> Import
  2. Browse to the dashboard that should be imported, e.g. Cinnamon-Actor-Dashboard.json, and click on OK.
  3. Click Save (the “disc” icon)
  4. Rinse and repeat for other dashboards you want to import

The dashboards are editable by default, i.e. anyone can add anything to them. To disable this functionality:

  1. Open a dashboard, e.g. Cinnamon Actors
  2. Click on the “wheel” symbol and then Settings
  3. Uncheck Editable under the Toggles menu
  4. Click Save
  5. Repeat for the other dashboards

The next step is to populate the database with data so that the dashboards are filled with metrics.

Wiring it together

Since this example uses StatsD as the metrics aggregator you must ensure that Lightbend Telemetry uses the StatsD plugin. The default settings are to publish to 127.0.0.1:8125.

Start your instrumented application with the StatsD plugin as reporter and data should flow into the Grafana dashboard. Allow for a couple of seconds to pass before you expect data to show up in the dashboard(s), the default frequency is set to publish into StatsD every 10s.

Make sure that you are looking at a dashboard that should receive any data! By default, nothing is instrumented, and you must configure what to instrument.

Dashboard details

This section describes the settings and metrics available in the dashboards. Every dashboard has a filter mechanism on the top of the dashboard with which it is possible to drill down to specific metric information. All filters support multiple selects. The following filters are available (where they make sense):

  • Servers : server the data originates from
  • Apps : application the data originates from
  • Actors : actor the data originates from
  • MetricValue : specific values
  • HeapType : JVM specific
  • MemoryPools : JVM specific
  • StopwatchIdentifiers : the stopwatch the data originates from

Detailed information about the StatsD wire can be found here.

Actor dashboard

The following metric information is available on the actor dashboard:

  • mailbox size
  • mailbox time
  • processed messages: 1 minute rate
  • processing time
  • sent messages: 1 minute rate
  • running actors
  • stashed message size per actor

Event dashboard

The following metric information is available on the event dashboard:

  • failure events (exceptions)
  • dead letters
  • unhandled messages
  • log error events
  • log warning events
  • mailbox size limit thresholds
  • mailbox time limit thresholds
  • processing time limit thresholds

Stopwatch dashboard

  • Stopwatch time
  • Stopwatch stop events per second: 1 minute rate rate
  • Stopwatch time threshold breaches

JVM dashboard

  • heap information
  • non-heap information
  • total heap
  • memory pools information
  • garbage collection time
  • garbage collection count
  • class loading

Lagom circuit breaker dashboard

  • status per circuit breaker: closed, open, half-open
  • throughput: mean rate
  • throughput: 15 minutes rate
  • throughput: 1 minute rate
  • success count
  • failure count