Using Google Cloud Pub/Sub as message broker

To configure access to your Google Cloud Pub/Sub broker for your Kalix project, you need to create a Google service account with access to your Google Cloud Pub/Sub broker and provide it to Kalix.

Details on doing this can be found in the Google documentation. We provide simplified steps below.

The service account should allow for the roles/pubsub.editor role.

Setting up the service account

To set up a service account and generate the key, follow these steps:

Navigate to https://console.cloud.google.com/.
From the blue bar, click the dropdown menu next to Google Cloud Platform.
Click New Project to create a project and save the <gcp-project-id>, which you will need later.

Enter the following gcloud command to set up the gcloud environment:

gcloud auth login
gcloud projects list
gcloud config set project <gcp-project-id>

Enter the following command to create the service account. The example uses the name kalix-broker, but you can use any name.
```
gcloud iam service-accounts create kalix-broker
```

Enter the following commands to grant the GCP Pub/Sub editor role to the service account. Substitute your project ID for <gcp-project-id>.

gcloud projects add-iam-policy-binding <gcp-project-id> \
    --member "serviceAccount:kalix-broker@<gcp-project-id>.iam.gserviceaccount.com" \
    --role "roles/pubsub.editor"

Generate a key file for your service account:

gcloud iam service-accounts keys create keyfile.json \
    --iam-account kalix-broker@<gcp-project-id>.iam.gserviceaccount.com

Now you have a service account key file with which to configure Kalix to use your Google Cloud Pub/Sub broker. You can add the key file using either the Kalix Console or the Kalix CLI.

Browser

Open the project in the Kalix Console.
Select Integrations from the left-hand navigation menu.
Click + for the Google Cloud Pub/Sub integration option.
Copy the contents of keyfile.json into the editor and, click Apply.

The project is now configured to use Google Pub/Sub as the message broker.

CLI

kalix projects config set broker \
  --broker-service google-pubsub \
  --gcp-key-file keyfile.json \
  --description "Google Pub/Sub in <gcp-project-id>"

The project is now configured to use Google Pub/Sub as the message broker.

Create a topic

To create a topic, you can either use the Google Cloud Console, or the Google Cloud CLI.

Browser

Open the Google Cloud Console.
Go to the Pub/Sub product page.
Click CREATE TOPIC on the top of the screen.
Fill in the Topic ID field and choose any other options you need.
Click CREATE TOPIC in the modal dialog.

You can now use the topic to connect with Kalix

Google Cloud CLI

gcloud pubsub topics create TOPIC_ID

You can now use the topic to connect with Kalix

Delivery characteristics

When your application consumes messages from Google Pub/Sub, it will try to deliver messages to your service in 'at-least-once' fashion while preserving order.

the GCP 'Subscription' has the 'Message ordering' flag enabled (this is the case by default for the subscriptions created by Kalix)
the code that acts as a publisher has 'message ordering' enabled (if needed on this client SDK)
an ordering key is provided for each message

When passing messages to a certain entity or using them to update a view row by specifying the id as the Cloud Event ce-subject attribute on the message, the same id must be used for the Google Pub/Sub ordering key to guarantee that the messages are processed in order by the entity or view.

Correct ordering is especially important for topics that stream directly into views using the transform_update option: when messages for the same subject id are spread over different ordering keys (or do not have ordering keys), they may read stale data and lose updates.

To achieve at-least-once delivery, messages that are not acknowledged before the Ack deadline will be redelivered. This means redeliveries of 'older' messages may arrive behind fresh deliveries of 'newer' messages.

When publishing messages to Google Pub/Sub from Kalix, the ce-subject attribute, if present, is used as the ordering key for the message.

Testing Kalix eventing

See Testing Kalix eventing.