Kafka - Amazon MSK

Amazon Managed Streaming for Apache Kafka new tab (Amazon MSK) is a fully managed Apache Kafka service.

Create Amazon MSK cluster

Create a new Kafka cluster from Amazon MSK console new tab. Select the region you are working in.

Select the following aside from defaults:

  • Create cluster with custom settings

  • VPC: Use the same as your EKS cluster is running in

  • Select the private subnets from the EKS VPC for each zone

  • Both TLS encrypted and plaintext traffic allowed

A typical cluster takes up to 15 minutes to create.

You find more detailed instructions and alternatives in the Amazon MSK documentation new tab

Allow EKS security group

To allow the nodes in the EKS cluster to connect to the MSK cluster you have to add a rule in the security group.

Go to the VPC console new tab. Select "Security Groups".

There are 3 security groups for the EKS cluster and you should select the one with description "EKS created security group …​". The one that has a name that doesn’t contain ControlPlaneSecurityGroup and doesn’t contain ClusterSharedNodeSecurityGroup . Make a note of this security group id for the EKS cluster.

Go back to the Amazon MSK console new tab and click on the "Security groups applied" in the MSK Cluster summary.

Edit inbound rules > add rule > All traffic > Source custom. Add the security group for the EKS cluster. Save rules.

Create topic

To create a topic or other administrative tasks you can connect to the MSK cluster from a Pod in your EKS cluster.

You can create a temporary Pod for this with:

kubectl run -i \
    --tty msk-mgmt \
    --image=adoptopenjdk:11-jre-hotspot \
    --restart=Never \
    --rm -- bash

Then install the Kafka command line tools with:

apt-get update
apt-get install wget -y
wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.12-2.8.0.tgz
tar -xzf kafka_2.12-2.8.0.tgz
cd kafka_2.12-2.8.0

From Amazon MSK console new tab you find the connection URL in the "View client information". Copy the list of Broker host/port pairs.

Save the connect string in an environment variable BOOTSTRAP_SERVERS and create a Kafka shopping-cart-events topic with:

BOOTSTRAP_SERVERS="<copied Bootstrap servers connect string>" && \
kubectl run -i --tty kafka-mgmt --image=confluentinc/cp-kafka --restart=Never --rm -- \
    kafka-topics \
    --bootstrap-server="$BOOTSTRAP_SERVERS" \
    --create \
    --topic shopping-cart-events \
    --replication-factor 2 \
    --partitions 4

Kafka configuration

Place the connection credentials in a Secret new tab. The Secret must contain:

  • bootstrapServersKey - the URL to the Kafka bootstrap servers

From Amazon MSK console new tab you find the connection URL in the "View client information". Copy the plaintext bootstrap servers.

The Secret can be created with for example:

kubectl create secret generic \
    shopping-cart-service-kafka-secret \
    --from-literal=bootstrapServers=<copied bootstrap servers connect string>

To enable the Kafka integration you define the name of the secret in kafka of the deployment descriptor:

apiVersion: akka.lightbend.com/v1
kind: AkkaMicroservice
metadata:
  name: shopping-cart-service
spec:
  image: <image>
  kafka:
    credentialsSecret: shopping-cart-service-kafka-secret

Apply the deployment descriptor:

kubectl apply -f kubernetes/shopping-cart-service-cr.yml

The Akka Operator will automatically provide the configuration for the connection based on the Secret when the application starts the ActorSystem.