Lagom configuration

Lagom endpoints need to be selected for Telemetry using configuration. Cinnamon accesses the same configuration as used to configure an ActorSystem. The default configuration is in application.conf.

Server metrics

Server metrics are automatically enabled when Cinnamon’s Lagom dependency is on the classpath.

It is possible to turn server-metrics off for a server with configuration, see Example configuration 3 here below.

Server metrics configuration

Endpoint metrics are not enabled by default. In other words, without any specific configuration, there will be no metrics for the endpoints.

Endpoint configuration settings are grouped per “host:port.” Cinnamon will inquire the configuration to determine how to set things up.

The configuration should define the following settings:

Setting Explanation
host:port The IP address and port the setting is valid for. Can be set to "*:*" to be valid for all hosts/ports.
server-metrics An optional setting (by default set to on) that can be used to turn metrics off for server(s).
paths A section in which endpoint paths are defined with specific settings per endpoint.

Each endpoint in the paths section should define the following:

Setting Explanation
endpoint path The endpoint path to configure. A * wildcard can be used at the end to select all subpaths.
metrics Should be set to on, true or yes for metrics to be created.

Server endpoint naming

By default, endpoint metrics created use the endpoint path as name for the metric. Let’s look at an example to clarify this feature:

Path directive Service definition Metric name
pathCall("/api/hello/:id", hello _) def hello(id: String) /api/hello/_id
pathCall("/order/:orderId/item/:itemId", getItem _) def getItem(orderId: Long, itemId: String) /order/_orderId/item/_itemId

Configuration conflict resolution

If there are several HTTP related configuration settings available on the classpath, e.g. Lagom and Akka HTTP, Lightbend Telemetry will merge all settings if possible. Should there be conflicting settings, the following order is applied: Lagom > Play > Akka HTTP.

Example configurations

Example configuration 1

The simplest configuration possible is the one that matches all paths, hosts and ports:

cinnamon.lagom.http.servers {
  "*:*" {
    paths {
      "*" {
        metrics = on
      }
    }
  }
}

Example configuration 2

Below is an example configuration that is more specific than Example configuration 1:

cinnamon.lagom.http.servers {
  "192.168.10.1:8080" {
    paths {
      "users/us/*" {
        metrics = on
      }
      "users/se/*" {
        metrics = on
      }
      "users/nz/*" {
        metrics = on
      }
    }
  }
}

The above configuration will create metrics for paths that map to users in us, se, or nz. Notice that all users in these domains are valid since we use * as part of the paths. Furthermore, this configuration is only valid for a server started on IP 192.168.10.1 and port 8080.

Example configuration 3

HTTP server configuration is selected from most specific to least specific port and host combination. This means that you can override wildcard configurations by having a more specific configuration section that have different settings.

Turn off all metrics for a particular host while still having them turned on for all other hosts:

cinnamon.lagom.http.servers {
  "*:*" {
    paths {
      "*" {
        metrics = on
      }
    }
  },
  "192.168.0.1:*" {
    server-metrics = off
  }
}

The server-metrics setting is optional and if not present it will automatically be set to on.

Client metrics configuration

Client metrics configuration settings are grouped per “host:port”. In the context of a Lagom service client we refer to “host:port” as a service. Each service can define which “paths” should generate metrics.

Let us look at a couple of client metrics configuration examples.

Example configuration 1

The simplest configuration possible is the one that matches all paths, hosts and ports:

cinnamon.lagom.http.clients {
  "*:*" {
    paths {
      "*" {
        metrics = on
      }
    }
  }
}

With the above configuration all service calls will be monitored and metrics generated will be under the service name “:”.

Example configuration 2

In the configuration below we use the setting service-name which means that metrics will be generated in the context of “accountService” and “customerService”.

cinnamon.lagom.http.clients {
  "192.168.10.1:*" {
    service-name = "accountService"
    paths {
      "*" {
        metrics = on
      }
    }
  },
  "192.168.10.10:8080" {
    service-name = "customerService"
    paths {
      "*" {
        metrics = on
      }
    }
  }
}

Example configuration 3

It is also possible to define what paths of a service that should generate and contribute to the metrics.

cinnamon.lagom.http.clients {
  "192.168.10.1:8080" {
    paths {
      "customer/address/*" {
        metrics = on
      }
    }
  }
}

Note: On the contrary to server endpoint metrics, in which unique metrics are created per path endpoint, any defined paths in client metrics configuration contribute to one overall metrics per service.

Circuit breaker configuration

To wire up Lagom circuit breakers telemetry in your Java application you must add the following line to the configuration file:

lagom.spi.circuit-breaker-metrics-class = "cinnamon.lagom.CircuitBreakerInstrumentation"

For Scala, cinnamon.lagom.CircuitBreakerInstrumentation should be wired in your application. See the Scala example for details.

This instructs Lagom to provide data to Lightbend Telemetry rather than the default behavior. The information gathered include:

  • State of circuit breaker — closed, open, half-open

  • Latency

  • Throughput

  • Success and failure counts

Lagom circuit breakers

It is important to have an understanding of the circuit breakers in Lagom in order to interpret the data that is gathered. In Lagom each client talking to another service uses a circuit breaker. This basically means that the circuit breaker is on the client instead of the server (in a traditional sense). This means that if your code calls a service directly there will not be any information gathered about this call. Currently only service to service calls are instrumented. See the Lagom documentation for more information on service clients and circuit breakers.

The state of a circuit breaker is one of:

State Interpretation Data Representation
open something is wrong 1
half-open things are going from bad to good 2
closed everything is fine 3

The Data Representation in the above table is the number associated with a certain state, i.e. this is what comes out of Lightbend Telemetry to any reporter listening to the data.

Lagom persistent entities

Lagom Persistent Entities are Akka sharded persistent actors. You can enable actor telemetry for the underlying actors.

Note: Sharded actors are created under the /system/sharding guardian actor. Note that Lagom Persistent Entities share the same actor class PersistentEntityActor.

You may want to group the persistent entity actors for each type of persistent entity and provide a custom name for it, e.g.

cinnamon.akka {
  actors {
    "EntityFoo-Actors" {
      report-by = group
      includes = ["/system/sharding/EntityFooName/*"]
      excludes = ["akka.cluster.sharding.Shard"]
    }
  }
}

Note how we exclude shard actors from the group. It’s important because the shard actors and the persistent entity actors share the same root actor path.

And vice versa we can create a group of the shard actors only if needed:

cinnamon.akka {
  actors {
    "EntityFoo-Shards" {
      report-by = group
      includes = ["/system/sharding/EntityFooName/*"]
      excludes = [
        "com.lightbend.lagom.internal.scaladsl.persistence.PersistentEntityActor",
        "com.lightbend.lagom.internal.javadsl.persistence.PersistentEntityActor"
      ]
    }
  }
}

In this example we exclude both Lagom PersistentEntity actor classes.