JMX Importer

The JMX Importer enables fetching data out of JMX MBeans and create metrics out of these via configuration. This section contains information how to set things up.

Dependency

Add the JMX Importer dependency to your build:

sbt
libraryDependencies += Cinnamon.library.jmxImporter
Maven
<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-jmx-importer</artifactId>
    <version>2.11.3</version>
</dependency>
Gradle
dependencies {
    compile group: 'com.lightbend.cinnamon', name: 'cinnamon-jmx-importer', version: '2.11.3'
}

Configuration

To configure the JMX Importer we must, apart from defining its context cinnamon.jmx-importer, provide settings for beans.

Required

No required configuration is needed for the importer to load, but without any beans definition there won’t be any metrics created.

Reference
cinnamon {
  jmx-importer {

    # The name keys to search for in the query string.
    #
    # Expected format: "<DOMAIN>:type=someType,<QUERY-NAME-KEY>=someName"
    # By default the following two formats will be searched for:
    # "<DOMAIN>:type=someType,name=someName"
    # "<DOMAIN>:type=someType,client-id=someName"
    #
    # Add any required identifiers that are needed to the list below.
    query-name-keys = ["name", "client-id"]

    identity {
      domain {
        category = "domains"
        key = "domain"
      }

      type {
        category = "types"
        key = "type"
      }

      name {
        category = "names"
        key = "name"
      }

      attribute {
        category = "attributes"
        key = "attribute"
      }
    }

    descriptor {
      group-name = "metrics"
    }
  }
}

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

The beans setting

The beans setting is where we define what JMX beans to retrieve and what attributes of them that should be used to create metrics. This section consists of a couple of subparts:

  • query: query used to retrieve the beans from JMX
  • suppress-log-warnings: indicates if any warnings, when getting JMX info for this bean, should be suppressed - optional and defaults to false
  • attributes: attributes that should be converted to metrics. Each attribute must contain the following:
    • attribute: name of the attribute
    • type: type of metrics that should be generated
  • additional-identity-keys: additional identities to include in final output - optional and defaults to []. Each identity must contain the following:
    • category: The category applied to the identity
    • key: The key which should be included. i.e. <context>:type=<someType>,name=<someName>,**additionalKey=<someValue>**

The JMX Importer currently supports two types of metrics:

  • GAUGE_LONG: generates a com.lightbend.cinnamon.metric.ProvidingGaugeLong.
  • GAUGE_DOUBLE: generates a com.lightbend.cinnamon.metric.ProvidingGaugeDouble.

Alternative query name format

The query format normally follows the pattern <context>:type=<someType>,name=<someName> but some clients, e.g. Kafka, use client-id instead of name. The Cinnamon JmxImporter is configurable to look for any key defined in the setting cinnamon.jmx-importer.query-name-keys. By default it will look for name and fall back to client-id if the former is not found. It is also possible to define custom keys. For more information see the “Reference” tab in the configuration above.

Extending the settings

Should you want to add more MBeans in your configuration, this can be done in the following way:

cinnamon.jmx-importer.beans = ${cinnamon.jmx-importer.beans} [
  // your bean definition(s) 
]

Example configurations

Let us take a look at some examples to get a better idea of what things can look like.

Simple

A simple configuration with which we get all beans in the domain someDomain, with type *, and name *. * represents wildcard, i.e. match everything in that context. Each bean matching is expected to have the attribute SomeAttribute and the type for this attribute is a long.

cinnamon.jmx-importer {
  beans = [
    {
      query = "someDomain:type=*,name=*",
      suppress-log-warnings = yes,
      attributes = [
        {
          attribute = "SomeAttribute",
          metric-type = "GAUGE_LONG"
        }
      ]
    }
  ]
}

JVM

An example of how to use the JMX Producer to generate some JVM related metrics in Cinnamon.

cinnamon.jmx-importer {
  beans = [
    {
      query = "java.lang:type=GarbageCollector,name=*"
      attributes = [
        {
          attribute = "CollectionCount",
          metric-type = "GAUGE_LONG"
        },
        {
          attribute = "CollectionTime",
          metric-type = "GAUGE_LONG"
        }
      ]
    },
    {
      query = "java.lang:type=OperatingSystem"
      attributes = [
        {
          attribute = "SystemCpuLoad",
          metric-type = "GAUGE_DOUBLE"
        }
      ]
    }
  ]
}

The above configuration will generate four unique metrics in Cinnamon.

Additional Identity Information

An example of how to output additional identity information beyond the defaults (domain/type/name/attributes). This example will include the partition and topic keys, which are useful for filtering purposes.

cinnamon.jmx-importer {
  beans = [
    {
      query = "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,partition=*,topic=*",
      attributes = [
        {
          attribute = "Count",
          metric-type = "GAUGE_DOUBLE"
        }
      ]
      additional-identity-keys = [
        {
          category = "partitions"
          key = "partition"
        },
        {
          category = "topics"
          key = "topic"
        }
      ]
    }
  ]
}

Output format

The output format depends on how metrics are reported, i.e. what reporter(s) that have been configured to be used. Whatever reporter used, the idea is to mimic the layout of the MBeans explorer in Java Mission Control. As an example, using the JVM configuration here above would generate the following format:

StatsD

Metrics output:

servers.<your server>.apps.<your app>.metrics.jmx.domains.java_lang.types.GarbageCollector.names.PS_MarkSweep.attributes.CollectionCount.CollectionCount
servers.<your server>.apps.<your app>.metrics.jmx.domains.java_lang.types.GarbageCollector.names.PS_MarkSweep.attributes.CollectionTime.CollectionTime
servers.<your server>.apps.<your app>.metrics.jmx.domains.java_lang.types.GarbageCollector.names.PS_Scavange.attributes.CollectionCount.CollectionCount
servers.<your server>.apps.<your app>.metrics.jmx.domains.java_lang.types.OperatingSystem.attributes.SystemCpuLoad.SystemCpuLoad

Elasticsearch

Metric output for one of the configured beans:

{
  ... some metadata
  "application" : <your app>,
  "attribute" : "SystemCpuLoad",
  "domain" : "java.lang",
  "host" : <your host>,
  "metric" : "attributes.SystemCpuLoad",
  "type" : "OperatingSystem"
}

Predefined JMX importers

Lightbend Telemetry comes with some predefined importers to provide for easy import from JMX.

HikariCP importer

HikariCP is a recommended connection pool framework that is used in Play.

HikariCP dependency

Add the HikariCP JMX Importer dependency to your build:

sbt
libraryDependencies += Cinnamon.library.hikariCPJmxImporter
Maven
<dependency>
    <groupId>com.lightbend.cinnamon</groupId>
    <artifactId>cinnamon-hikaricp-jmx-importer</artifactId>
    <version>2.11.3</version>
</dependency>
Gradle
dependencies {
    compile group: 'com.lightbend.cinnamon', name: 'cinnamon-hikaricp-jmx-importer', version: '2.11.3'
}

Note: Make sure that HikariCP is configured to register MBeans by setting the registerMbeans setting to true for HikariCP.

HikariCP configuration

Required

No required configuration is needed for the HikariCP JMX importer to import the MBeans.

Reference
query = "com.zaxxer.hikari:type=Pool *"
attributes = [
  {
    attribute = "ActiveConnections",
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = "IdleConnections",
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = "ThreadsAwaitingConnection",
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = "TotalConnections",
    metric-type = "GAUGE_LONG"
  }
]
query = "com.zaxxer.hikari:type=PoolConfig *"
attributes = [
  {
    attribute = "ConnectionTimeout",
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = "IdleTimeout",
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = "LeakDetectionThreshold",
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = "MaxLifetime",
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = "MaximumPoolSize",
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = "MinimumIdle",
    metric-type = "GAUGE_LONG"
  },
  {
    attribute = "ValidationTimeout",
    metric-type = "GAUGE_LONG"
  }
]

Note: These settings are defined in the reference.conf. You only need to specify any of these settings when you want to override the defaults.

It is possible to add more MBeans by extending the settings if there is something missing.