Lagom

As of version 1.4 Lightbend’s Lagom framework, Lightbend Telemetry has support for both server and client metrics. Also, Lagom microservices contains built-in circuit breakers that can be instrumented with Lightbend Telemetry.

Note: Lightbend Telemetry will not collect any metrics in Lagom development mode, e.g., when using runAll. The reason for this is that runAll does not fork the JVM when running and a forked JVM is something the Cinnamon agent needs to be able to instrument the Akka code.

Cinnamon Lagom module dependency

Add the Cinnamon Lagom dependency to your build file:

sbt
libraryDependencies += Cinnamon.library.cinnamonLagom
Maven
<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-lagom_2.11</artifactId>
    <version>2.14.1</version>
</dependency>
Gradle
dependencies {
  compile group: 'com.lightbend.cinnamon', name: 'cinnamon-lagom_2.11', version: '2.14.1'
}

Lagom server metrics

  • Connections (rate) — the rate of connections per server.

  • Requests (rate) — the rate of requests per server.

  • Responses (rate) — the rate of responses per server.

  • Responses 2xx (rate) — the rate of responses with 200-299 status codes per server.

  • Responses 3xx (rate) — the rate of responses with 300-399 status codes per server.

  • Responses 4xx (rate) — the rate of responses with 400-499 status codes per server.

  • Responses 5xx (rate) — the rate of responses with 500-599 status codes per server.

  • Response time (recorder) — the time for responses per server.

  • Response time 2xx (recorder) — the time for responses with 200-299 status codes per server.

  • Response time 3xx (recorder) — the time for responses with 300-399 status codes per server.

  • Response time 4xx (recorder) — the time for responses with 400-499 status codes per server.

  • Response time 5xx (recorder) — the time for responses with 500-599 status codes per server.

Note: Server metrics are created automatically when the Cinnamon Lagom module dependency is added. It is possible to turn server metrics off via configuration.

Lagom server endpoint metrics

The following metrics are recorded for configured endpoints.

  • Endpoint responses (rate) — the rate of responses per endpoint.

  • Endpoint responses 2xx (rate) — the rate of endpoint responses with 200-299 status codes (optional).

  • Endpoint responses 3xx (rate) — the rate of endpoint responses with 300-399 status codes (optional).

  • Endpoint responses 4xx (rate) — the rate of endpoint responses with 400-499 status codes (optional).

  • Endpoint responses 5xx (rate) — the rate of endpoint responses with 500-599 status codes (optional).

  • Endpoint response time (recorder) — the time for responses per endpoint.

  • Endpoint response time 2xx (recorder) — the time for endpoint responses with 200-299 status codes (optional).

  • Endpoint response time 3xx (recorder) — the time for endpoint responses with 300-399 status codes (optional).

  • Endpoint response time 4xx (recorder) — the time for endpoint responses with 400-499 status codes (optional).

  • Endpoint response time 5xx (recorder) — the time for endpoint responses with 500-599 status codes (optional).

Note: By default, no metrics are created for endpoints. Lagom Configuration is used to instruct Cinnamon what endpoints to create metrics for. Endpoint metrics per status code class can be enabled optionally.

Lagom client metrics

The following metrics are recorded for instrumented Play WS clients, metric in parentheses:

Client metrics

  • Requests (rate) — the rate of requests from a client to a service.

  • Service response time (recorder) — the time for responses from a service.

Note: By default, no metrics are created for endpoints. Lagom Configuration is used to instruct Cinnamon what endpoints to create metrics for.

Lagom persistence metrics

Lagom persistent entities are built on top of Akka Persistence. Thus all the metrics and events provided by Akka Persistence are applicable for Lagom persistent entities. For how to enable it and what metrics and events are provided see the Akka Persistence section.

Also see Persistence Entity Configuration for Lagom persistent entities configuration specifics.

Lagom Projection metrics

Starting in Lagom 1.6.2 Lightbend Telemetry can record metrics for Lagom Projections (Read Side Processors and Topic Producers). All metrics are tagged with both the name of the projection and the tag the persistent entity added to the event.

The following metrics are recorded for projections.

  • Event processed successfully (count) — number of events processed successfully.

  • Event process failures (rate) — events process failures.

  • Query Latency (recorder) — the time (in milliseconds) from the moment the event was created to the moment it was read to be processed. Multiple Akka Persistence plugins add a fixed query latency to prevent data loss when querying caused by clock skew in the various nodes writing to the database journal.

  • Processing Time (recorder) — the time (in milliseconds) it took to process the event.

  • Latency (recorder) — the time (in milliseconds) from the moment the event was created to the moment it is considered as fully processed (which includes updating the offset store in all cases).

Note: Due to differences in the two projection implementations in Lagom, Processing Time means slightly different things depending on the projection being a Read Side Processor or a Topic Producer. In Read Side Processor, Processing Time records the period of time in the user event handler plus the time to execute any database operation (including the update of the offset in the database). In Topic Producers, Processing Time records only the period of time in the event handler but not the network round-trip to the Broker.

Note: The Latency metric records a time period which includes the period measured by Query Latency and the period measured by Processing Time and some more operations which may not be individually recorded by Lightbend Telemetry.

Lagom Projections configuration provides more details on configuring projection-related metrics.

Clock Skew

An event processed by a projection may have been created on a different machine. Latency, for example, measures the time period between the moment the event was created, and the moment the projection completed successfully.

As a consequence, given the skew between the clocks of different machines some of the metrics above will only be reported partially. Especifically, negative time periods will not be recorded. Non-negative values, though, will be recorded but may be impacted by clock skew so the data displayed will be of poor quality.

Lagom persistent entities tracing

Enabing tracing for Lagom persistent entities is equivalent to enabling Akka Persistence tracing. Akka Persistence tracing enabling for more details.

Lagom request tracing

Request tracing is supported through the Cinnamon OpenTracing integration. See Lagom configuration for OpenTracing.

Ask-pattern metrics

Lagom persistent metrics can be configured to collect ask-pattern metrics for each persistent entity type. See ask-pattern sharded entities configuration for the details.