Kafka - Strimzi operator

For production usage we recommend using Amazon MSK or Confluent Cloud (Kafka-as-a-Service).

For development or test environments you may want to manage your own Kafka cluster with Strimzi. Strimzi new tab simplifies the process of running Apache Kafka in a Kubernetes cluster.

Install Strimzi

kubectl create namespace akka-demo

Add the Strimzi Helm repository and update the local index.

helm repo add strimzi https://strimzi.io/charts/
helm repo update

Install the latest version of Strimzi.

helm install strimzi strimzi/strimzi-kafka-operator --namespace akka-demo

After the install completed, the Strimzi Kafka operator should be running in the given namespace.

kubectl config set-context --current --namespace=akka-demo
kubectl get pods

To create a Kafka cluster using Strimzi, create the custom resource for the Strimzi Kafka operator.

Create a file kubernetes/strimzi-cr.yml according to one of the examples in strimzi-kafka-operator/examples/kafka/ new tab. See configuration options in the Strimzi documentation new tab.

Note that the topicOperator should be included to be able to Create topic.

Apply the kubernetes/strimzi-cr.yml to your Kubernetes cluster:

kubectl apply -f kubernetes/strimzi-cr.yml

Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the Kafka resource when all pods are ready:

kubectl describe kafka kafka-strimzi

kubectl get kafka kafka-strimzi -o=jsonpath='{.status.listeners[?(@.type=="plain")].bootstrapServers}{"\n"}'

Create topic

The Strimzi operator can create topics from a KafkaTopic custom resource. It’s important that the custom resource for the Strimzi Kafka operator included the topicOperator. See configuration options in the Strimzi documentation new tab.

kubernetes/strimzi-topic-cr.yml
apiVersion: kafka.strimzi.io/v1beta1
 kind: KafkaTopic
 metadata:
   name: shopping-cart-events
   labels:
     strimzi.io/cluster: kafka-strimzi
 spec:
   topicName: shopping-cart-events
   partitions: 4
   replicas: 2
kubectl apply -f kubernetes/strimzi-topic-cr.yml

Configure connection

Place the connection credentials in a Secret new tab. The Secret must contain:

  • bootstrapServersKey - the URL to the Kafka bootstrap servers

The Secret can be created with for example:

kubectl create secret generic \
    shopping-cart-service-kafka-secret \
    --from-literal=bootstrapServers=<bootstrap servers connect string>

Use the bootstrap servers retrieved above.

To enable the Kafka integration you define the name of the secret in kafka of the deployment descriptor:

apiVersion: akka.lightbend.com/v1
kind: AkkaMicroservice
metadata:
  name: shopping-cart-service
spec:
  image: <image>
  kafka:
    credentialsSecret: shopping-cart-service-kafka-secret

Apply the deployment descriptor:

kubectl apply -f kubernetes/shopping-cart-service-cr.yml

The Akka Operator will automatically provide the configuration for the connection based on the Secret when the application starts the ActorSystem.

Uninstall Strimzi

If you want to remove Strimzi, you can do that with the following Helm command:

helm delete strimzi -n akka-demo