SLF4J MDC

Lightbend Telemetry supports propagation of the SLF4J Mapped Diagnostic Context (MDC) through Akka actors, Akka Scheduler (scheduleOnce), Akka Streams, Akka HTTP requests, Scala Futures, and Java Futures. When an MDC is set on the current thread, this will be copied with any subsequent message sends, HTTP requests, or asynchronous callbacks. The MDC will be reinstated when the receiving actor processes the message, the server receives the request, the client receives the response, or when the callback runs. For example, this allows a request or transaction ID to be set and to appear in all connected log messages.

Cinnamon dependency

First make sure that your build is configured to use the Cinnamon Agent.

To enable SLF4J MDC support, add the following dependency to your build:

sbt

copylibraryDependencies += Cinnamon.library.cinnamonSlf4jMdc

Maven

copy<dependency>
  <groupId>com.lightbend.cinnamon</groupId>
  <artifactId>cinnamon-slf4j-mdc_2.13</artifactId>
  <version>2.20.3</version>
</dependency>

Gradle

copydependencies {
  implementation group: 'com.lightbend.cinnamon', name: 'cinnamon-slf4j-mdc_2.13', version: '2.20.3'
}

No further configuration is required. All actors (including the Akka logging actors and remote actors), Akka Streams, Akka HTTP servers and clients, and Scala Future or Java Future callbacks will transport and receive the current MDC.

MDC filter

The MDC keys that get transferred across asynchronous boundaries can be filtered using configuration. For example, use the following configuration to only propagate the "id" value:

copycinnamon.slf4j.mdc {
  filter = ["id"]
}

MDC filter for HTTP requests and responses

A filter only for HTTP client requests and server responses can be configured separately, so that any sensitive information in the MDC can be propagated locally but won’t be included in HTTP headers. Use the http.filter setting to configure which MDC keys should be propagated with requests and responses:

copycinnamon.slf4j.mdc {
  http.filter = ["Correlation-ID"]
}

Filters can also be specified per-endpoint as an extension of the HTTP configuration for Akka HTTP using the mdc.filter setting:

copycinnamon.akka.http {
  clients {
    "*:*" {
      paths {
        "hello/*" {
          mdc.filter = ["Correlation-ID"]
        }
      }
    }
  }
  servers {
    "*:*" {
      paths {
        "hello/*" {
          mdc.filter = ["Correlation-ID"]
        }
      }
    }
  }
}

MDC filter for Alpakka Kafka

A filter only for Alpakka Kafka producers can be configured separately, so that any sensitive information in the MDC can be propagated locally but won’t be included in Kafka record headers. Use the alpakka.kafka.filter setting to configure which MDC keys should be propagated with producer records:

copycinnamon.slf4j.mdc {
  alpakka.kafka {
    filter = ["Correlation-ID"]
  }
}

Serialization limit

The MDC is transported along with remote messages and requests. To avoid inadvertently sending a lot of data, there is a limit on the size of the serialized MDC. If the limit is exceeded then the MDC will not be propagated remotely and a warning will be logged. If you reach this limit, you may want to only transport specific keys using the MDC filter configuration, or the limit can be increased. For example, you can use the following configuration to set a different limit:

copycinnamon.slf4j.mdc {
  serialization.limit = 2048 bytes
}

Akka HTTP headers

For Akka HTTP requests and responses, the MDC is propagated as HTTP message headers. There are two ways to propagate the MDC in headers: either as a single encoded header, or as individual headers for each MDC value with a prefixed key used as the header name.

Akka HTTP encoded header

The default MDC propagation for Akka HTTP is as a single encoded header, where the MDC context map is serialized and then encoded in Base64. For example, an MDC that contains:

copy"Correlation-ID" -> "abc123"

would be propagated with this header:

copyCinnamon-MDC: AAAAAQAAAA4AAAAOQ29ycmVsYXRpb24tSUQAAAAGYWJjMTIz

The header name is Cinnamon-MDC by default and can be configured. For example, the following configuration sets a different header name:

copycinnamon.slf4j.mdc {
  http {
    encoded-header {
      name = "X-MDC"
    }
  }
}

The filter and serialization limit are also applied to the encoded MDC header.

Akka HTTP mapped headers

The MDC propagation for Akka HTTP also supports individual headers, where each MDC value has its own header with a prefixed key. For example, an MDC that contains:

copy"Correlation-ID" -> "abc123"

would be propagated with this header:

copyCNMDC-Correlation-ID: abc123

To enable mapped headers, set the propagation method using this configuration:

copycinnamon.slf4j.mdc {
  http.propagation = mapped-headers
}

The prefix used for mapped headers is configurable. For example, the following configuration sets a different header prefix:

copycinnamon.slf4j.mdc {
  http {
    mapped-headers {
      prefix = "X-"
    }
  }
}

Alpakka Kafka headers

For Alpakka Kafka, the MDC is propagated as Kafka record headers. Similar to Akka HTTP headers, there are two ways to propagate the MDC in Kafka headers: either as a single encoded header, or as individual headers for each MDC value with a prefixed key used as the header name.

Alpakka Kafka encoded header

The default MDC propagation for Alpakka Kafka is as a single encoded header, where the MDC context map is serialized into bytes. The header name is Cinnamon-MDC by default and can be configured. For example, the following configuration sets a different header name:

copycinnamon.slf4j.mdc {
  alpakka.kafka {
    encoded-header {
      name = "X-MDC"
    }
  }
}

Alpakka Kafka mapped headers

The MDC propagation for Alpakka Kafka also supports individual headers, where each MDC value has its own header with a prefixed key and the value encoded as UTF-8 bytes. The default prefix is CNMDC-. To enable mapped headers, set the propagation method using this configuration:

copycinnamon.slf4j.mdc {
  alpakka.kafka {
    propagation = mapped-headers
  }
}

The prefix used for mapped headers is configurable. For example, the following configuration sets a different header prefix:

copycinnamon.slf4j.mdc {
  alpakka.kafka {
    propagation = mapped-headers
    mapped-headers {
      prefix = "X-"
    }
  }
}

Log correlation

Cinnamon supports adding correlation identifiers to the MDC for correlating causally related log messages across nodes or threads. Any correlation identifiers will be automatically propagated across asynchronous or remote boundaries with subsequent message sends, HTTP requests, or asynchronous callbacks. You can configure Cinnamon to include random correlation IDs automatically, or to include the current trace ID if OpenTracing is being used.

Automatic correlation IDs

Cinnamon supports automatically inserting random correlation IDs. For example: on HTTP server requests, if there is no correlation ID already in context, Cinnamon can add a random UUID to the MDC. The injected correlation ID will then be propagated with the MDC and can be included in all log messages associated with the initial request, including other HTTP requests, actors, streams, or futures. The correlation ID will also be automatically included for propagation if an MDC filter is configured.

To enable automatic correlation IDs use this configuration:

copycinnamon.slf4j.mdc {
  log-correlation += automatic-correlation-id
}

The correlation ID uses the MDC key Correlation-ID by default. If needed, you can configure the name used for the correlation ID:

copycinnamon.slf4j.mdc {
  log-correlation += automatic-correlation-id

  automatic-correlation-id {
    name = "Named-ID"
  }
}

By default, the automatic correlation ID is only added at HTTP entry points, and then propagated from there. To check and insert correlation IDs within all initial scopes, including initial Actor message sends, enable the all-scopes setting:

copycinnamon.slf4j.mdc {
  log-correlation += automatic-correlation-id

  automatic-correlation-id {
    all-scopes = on
  }
}

Normalizing correlation ID key names

Due to case-insensitive handling of HTTP header names, and case-sensitive handling of MDC keys, under default settings it is possible that a log correlation ID passing over such a boundary could be missed (e.g., if an HTTP header was specified as correlation-id: 123 but the expected MDC key was specifically Correlation-ID).

For these cases, the normalize-key setting can be enabled, causing Cinnamon to normalize the correlation key to the correct case-sensitive version within the MDC the first time it is encountered.

copycinnamon.slf4j.mdc {
  log-correlation += automatic-correlation-id

  automatic-correlation-id {
    normalize-key = true
  }
}

OpenTracing trace IDs

If the Cinnamon OpenTracing integration is being used, log correlation can be enabled for trace IDs. The current trace ID for sampled traces will be inserted into the MDC, to allow logs to be correlated with recorded traces. Note that events can also be logged into traces using active span logging.

To enable trace ID log correlation use this configuration:

copycinnamon.slf4j.mdc {
  log-correlation += opentracing-trace-id
}

The trace ID uses the MDC key Trace-ID by default. If needed, you can configure the name used for the trace ID:

copycinnamon.slf4j.mdc {
  log-correlation += opentracing-trace-id

  opentracing-trace-id {
    name = "Named-ID"
  }
}

Depending on the OpenTracing implementation, the trace ID can be in many shapes and forms. If you are running in an environment with multiple tracing implementations on different systems and need to ensure that the trace ID is zero padded to either 16 or 32 characters as defined by the B3 header specification, then you can set the padding to either 16 or 32:

copycinnamon.slf4j.mdc {
  log-correlation += opentracing-trace-id

  opentracing-trace-id {
    b3-zero-padding = 32
  }
}

By default, the trace ID is only added at HTTP entry points, and then propagated from there. To check and insert trace IDs within all initial scopes, including initial Actor message sends, enable the all-scopes setting:

copycinnamon.slf4j.mdc {
  log-correlation += opentracing-trace-id

  opentracing-trace-id {
    all-scopes = on
  }
}

By default, trace IDs will only be added for sampled traces — those traces which will be reported to the tracing backend. To include trace IDs for all traces, both sampled and non-sampled, disable the sampled-only setting:

copycinnamon.slf4j.mdc {
  log-correlation += opentracing-trace-id

  opentracing-trace-id {
    sampled-only = off
  }
}

Trace ID normalization

When using OpenTracing trace IDs for log correlation, key normalization can be enabled using a normalize-key setting similar to the one discussed here. The following example enables key normalization when using OpenTracing identifiers for log correlation:

copycinnamon.slf4j.mdc {
  log-correlation += opentracing-trace-id

  opentracing-trace-id {
    normalize-key = true
  }
}