Akka Persistence

Lightbend Telemetry is capable of capturing data for the following Akka Persistence related features for Akka 2.5 and above.

Cinnamon Akka Persistence module dependency

After adding the Cinnamon Agent as described in the setup, make sure that you add the Cinnamon Akka Persistence module dependency to your build file:

sbt
libraryDependencies += Cinnamon.library.cinnamonAkkaPersistence
Maven
<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-akka-persistence_2.12</artifactId>
  <version>2.12.2</version>
</dependency>
Gradle
dependencies {
  compile group: 'com.lightbend.cinnamon', name: 'cinnamon-akka-persistence_2.12', version: '2.12.2'
}

Persistence metrics

The following Persistence metrics are recorded, type of metric in parenthesis.

Persistent entity metrics:

  • Active entities (gauge) — number of active entities (in-memory, excludes passivated entities).
  • Received commands (gauge)* — number of received commands.
  • Command in-stash time (recorder)* — time that commands spend in the stash. See Akka Persistence internal stash for the details.
  • Command processing time (recorder)* — time for processing the command excluding time spent in the mailbox or stash.
  • Persisted events (gauge)* — number of persisted events acknowledged by the journal.
  • Event persistence time (recorder)* — time from the moment the persist/persistAll method was called to the moment when it was acknowledged by the journal.
  • Recovery time (recorder) — time from the moment the permit is granted to the moment recovery is complete; this includes fetching events and snapshots from the journal.
  • Recovery failure time (recorder) — failure recovery time for a persistent entity (in case of failure).

Recovery permit metrics:

When starting many persistent entites at the same time the journal data store is protected from being overloaded by limiting number of recoveries that can be in progress at the same time. The limit is controlled via configuration akka.persistence.max-concurrent-recoveries. See Akka Persistence recovery for the details.

  • Used permits (recorder) — number of used permits.
  • Pending permits (recorder) — number of entities waiting for a permit to start recovery.
  • Max permits (gauge) — max number of permits (set via Akka configuration).

The metrics marked with * support automatic command type or event type dimension. It’s disabled by default, see below for enabling command and event dimensions.

Dimensions

Opt-in dimensions are available that enable collecting metrics by command or event type. By default these dimensions are disabled but can be enabled for each selection separately:

command-type
cinnamon.akka {
  persistence.entities {
    "/user/*" {
      report-by = class
      command-type = on
    }
  }
}
event-type
cinnamon.akka {
  persistence.entities {
    "/user/*" {
      report-by = class
      event-type = on
    }
  }
}

Persistence events

The following Persistence events are generated:

  • Recovery failure — Event created whenever a message replay fails. Event information: actor-ref — the actor trying to recover failure — the exception that occurred event — the event that failed if available recovery-failure-time — the time in nanoseconds

  • Persist failure — Event created whenever a message persist fails. Event information: actor-ref — the actor failing to persist an event failure — the exception that occurred event — the event that failed if available sequence-number — the event sequence number

  • Persist rejection — Event created whenever a message persist is rejected. Event information: actor-ref — the actor failing to persist an event failure — the exception that occurred event — the event that failed if available sequence-number — the event sequence number

Detailed information

For specific information of how to configure persistent actors see: