Actor configuration

Actors need to be selected for telemetry using configuration. Cinnamon accesses the same configuration as used to configure an ActorSystem. The default configuration is in application.conf.

Note: Actors are not instrumented automatically. Actors must have instrumentation explicitly enabled - see sections below for how to.

Actor selection

To avoid any unwanted impact on application performance, you need to decide which actors in your system are instrumented. Actor selections are based on either the class of the actor, using fully qualified class name, package, or outer class, or the path of the actor, using full paths or subtrees.

Class selection

The following are valid actor selections using fully qualified class names:

"com.example.a.A"
"com.example.a.A$B"

Path selection

The following is a valid actor selection using the full actor path:

"/user/a/b"

Note that paths include the user guardian actor.

Wildcard selection

A * wildcard can also be used at the end of a selection to select all classes within the given package or class, or all instances within the given actor subtree. The following are valid actor selections for classes in a package, or for inner classes:

"com.example.a.*"
"com.example.a.A$*"

The following is a valid actor selection for actors within a subtree:

"/user/a/*"

Actor settings

Settings can be applied to each configured actor selection. The main setting is report-by, for enabling instrumentation and deciding how the selected actors will report metrics.

Report-by settings

Instrumentation is enabled for an actor selection using the report-by setting. There are two main ways to report metrics: either metrics are aggregated by actor class, or metrics are reported for individual actor instances using the actor path. Actors can be selected for configuration by either class or path, and then reported by either class or path.

Here’s an example configuration that enables instrumentation for several actors:

cinnamon.akka {
  actors {
    "com.example.a.A" {
      report-by = class
    }
    "com.example.a.b.*" {
      report-by = instance
    }
    "/user/x/y" {
      report-by = instance
    }
    "/user/x/y/*" {
      report-by = class
    }
  }
}

Note: Actors that are reported by instance should be carefully selected, to avoid too many unique metric identifiers. There is a hard limit on the number of metric identifiers that can be reported, to prevent overloading the monitoring system.

Actor excludes

Wildcards can be used to select all actor classes in a package or all actors in a subtree. Actors can be excluded from a selection using excludes. For example:

cinnamon.akka {
  actors {
    "com.example.a.*" {
      report-by = class
      excludes = ["com.example.a.B", "com.example.a.b.*"]
    }
    "/user/x/*" {
      report-by = instance
      excludes = "/user/x/y/z"
    }
  }
}

The above configuration enables telemetry for all actor classes in the com.example.a package, apart from the com.example.a.B actor class and those actor classes in the com.example.a.b sub-package. It enables instance reporting for all actors in the /user/x/* subtree, apart from the /user/x/y/z actor.

Routers

Metrics for routers are created specifically by default. This comes with an impact on the performance and should you want to minimize this impact it is possible to turn off the router metrics for a selection with the setting routers = off:

cinnamon.akka {
  actors {
    "com.example.a.b.*" {
      report-by = instance
      routers = off
    }
  }
}

Router configuration settings

Routers are unique in that they are created automatically by Akka. Consider the following example code:

// inside "myActor"
val router1: ActorRef = context.actorOf(RoundRobinPool(10).props(Props[SomeRoutee]), "r1")
//...
val router2: ActorRef = context.actorOf(RoundRobinPool(5).props(Props[AnotherRoutee]), "r2")

When setting up the configuration for the code above, it is important to understand the underlying mechanism in Akka. The router actor class used in Akka is akka.routing.RouterActor and this means that if you decide to use report-by=class all metrics for all routers in that actor system will be grouped together into this particular class. In other words, let’s say your actor system contains two routers, router1 and router2, and that you have report-by=class configured, this means that the metrics gathered will be merged into one class, i.e. akka.routing.RouterActor.

If you instead want to see metrics for each router then configure report-by=instance for each of them. In the case of instance, the metrics will be reported for each router instance, and each router actor path will be used as the identifier. In the example above this means that metrics will be generated for both /user/myActor/r1 and /user/myActor/r2.

Actor groups

Metrics can be grouped by actor class, as described above. Other groups can be formed using the group setting for report-by.

For example, the following configuration selects all actors in a package and then reports metrics for all these actors as a group, named by the package selection com.example.a.b.*:

cinnamon.akka {
  actors {
    "com.example.a.b.*" {
      report-by = group
    }
  }
}

The following configuration selects all actors in a subtree and reports metrics for these actors as a group, named by the subtree selection /user/x/y/*:

cinnamon.akka {
  actors {
    "/user/x/y/*" {
      report-by = group
    }
  }
}

Groups can have any name assigned. By default, the name and the actor selection are the same, but the name can also be defined separately by putting actor selections under includes.

For example, the following configuration selects all actors in either the com.example.a.b.* package or the /user/x/y/* subtree, and then reports metrics for these actors as a group named actor-group:

cinnamon.akka {
  actors {
    "actor-group" {
      report-by = group
      includes = ["com.example.a.b.*", "/user/x/y/*"]
    }
  }
}

Template groups

It is also possible to define groups to be created dynamically using actor path patterns containing a ?. These definitions must have the report-by set to group and no includes setting since that is automatically created.

The following configuration will create groups for every actor below the /user/e/ path and have all children of that actor report to the corresponding group:

cinnamon.akka {
  actors {
    "/user/e/?/*" {
      report-by = group
    }
  }
}

These groups can of course have the normal thresholds and excludes settings as well as special group relative excludes settings.

The following configuration will create groups for every actor below the /user/e/ path but exclude the path named /user/e/r2/*:

cinnamon.akka {
  actors {
    "/user/e/?/*" {
      report-by = group
      excludes = "./r2/*"
    }
  }
}

Note: Pattern groups will have a performance impact on actor creation, since every actor needs to be checked against the patterns to see if it will create a new group. Also creating and destroying actors that create a new group will repeatedly update shared data structures and impact performance even more.

Multiple report-by settings

Actors may match multiple actor selections and can report metrics under both class and instance. It’s also possible to specify multiple report-by settings for a single actor selection. For example, given the following configuration:

cinnamon.akka {
  actors {
    "com.example.a.*" {
      report-by = class
    }
    "com.example.a.b.*" {
      report-by = instance
    }
    "/user/x/y/*" {
      report-by = [class, instance]
    }
  }
}

An actor with the class com.example.a.b.C or the path /user/x/y/z would report metrics under both actor class and actor path.

Metric thresholds

Some metrics support configured thresholds. If the threshold is exceeded then an event is recorded. Threshold values are set under thresholds in the actor settings.

The following threshold settings are supported:

Setting Type Description
mailbox-size Integer Mailbox queue grows too large
stash-size Integer Stash queue grows too large
mailbox-time Duration Message has been in the mailbox for too long
processing-time Duration Message processing takes too long
remote-message-size Bytes Remote message is too big

Thresholds are only checked when an actor selection has a threshold enabled via configuration. Here is an example of adding thresholds to select actors:

cinnamon.akka {
  actors {
    "com.example.a.A" {
      report-by = class
      thresholds {
        mailbox-size = 1000
        stash-size = 50
        mailbox-time = 3s
        processing-time = 500ms
      }
    }
    "com.example.a.b.*" {
      report-by = class
      thresholds {
        mailbox-size = 100
        remote-message-size = 512K
      }
    }
    "/user/x/y" {
      report-by = instance
      thresholds {
        mailbox-time = 200ms
      }
    }
    "/user/x/y/*" {
      report-by = class
      thresholds {
        stash-size = 100
        processing-time = 1s
        remote-message-size = 1M
      }
    }
  }
}

Setting fallbacks

An actor class may match multiple actor selections in the configuration. Reporting settings will be merged together but other settings, like thresholds, will ‘fallback’ to higher matches. Specific actor settings will inherit the higher settings.

For example:

cinnamon.akka {
  actors {
    "com.example.a.*" {
      report-by = class
      thresholds {
        mailbox-size = 1000
        mailbox-time = 3s
        processing-time = 500ms
      }
    }
    "com.example.a.b.*" {
      report-by = class
      thresholds {
        mailbox-size = 100
        stash-size = 125
      }
    }
    "com.example.a.b.C" {
      report-by = class
      thresholds {
        mailbox-time = 1s
      }
    }
  }
}

Given the above configuration, the actor class com.example.a.b.C will have the following threshold settings:

mailbox-time = 1s        # provided explicitly
mailbox-size = 100       # inherited from "com.example.a.b.*"
stash-size = 125         # inherited from "com.example.a.b.*"
processing-time = 500ms  # inherited from "com.example.a.*"

Dimensions

When metrics and events are reported for actors, the identity of the actor will contain two dimensions: actor system and dispatcher. These may be represented in the metric key or in metric tags, depending on the backend used. Whether or not these dimensions are enabled can be configured using the dimensions setting. For example, here is the configuration to disable the grouping of actors by dispatcher and only use the actor system dimension:

cinnamon.akka {
  dimensions = [actor-system]
}

There is also an address dimension that is disabled by default for backwards compatibility reasons. This dimension represents the actor system address in the form host:port, with local being used if the actor system doesn’t have remoting enabled. To enable the address dimension simply add it in configuration like this:

cinnamon.akka {
  dimensions += address
}