Cluster Aware Actors

Akka Tip of the Month - July 2017

Hugh McKee, June 20, 2017

The fundamental building block of Akka actor systems is an actor. Actors are mechanically simple yet potentially very powerful. From the perspective of an actor client, when the client sends a message to an actor, the target actor is a black box. How a given actor handles messages is determined by how each actor is implemented. When an actor receives a message, it can ignore it, or it can perform some operation and then may or may not send a response message that the operation has been completed. An actor may delegate work out to other actors, and these other actors may process their messages concurrently.

The possibilities are almost limitless for implementing actors. In a way, actors are analogous to Lego or Minecraft blocks. With the simple building blocks of Lego and Minecraft, it is possible to craft magnificent structures. With the humble actor, you can craft amazing and powerful systems.

Adding to the basic capabilities of actors is the fact that the message senders and the receiving actors are not constrained to a single JVM process boundary. One of the best features of Akka is that you can build systems that can run in a cluster. An Akka cluster is a set of multiple nodes each running in independent JVMs. Programmatically it is just as easy to send a message to an actor in a local JVM as it is to send a message to an actor running in another JVM. As shown in Figure 1, actors distributed on multiple cluster nodes can send message to other actors on other cluster nodes.

actors in a cluster

Figure 1 - Actors in a cluster sending messages to each other

Running in a clustered environment adds an entirely new dynamic to the architecture of an actor system. It is one thing to be running on a single server, in a single process, and within a single JVM. It is entirely another thing to be running a system that spans a cluster of JVMs spread across a network.

In a single JVM, with actors running in an actor system, the JVM is either running or not running. On the other hand, when running in a cluster, at any point in time the topology of the cluster may change. Cluster node JVMs may come and go at a moment’s notice.

The cluster itself is technically is up as long as at least one node is up. Actors on one node may be happily exchanging messages with actors on other nodes then, without warning, a node goes away taking down the actors that were resident on that node. How are the remaining actors supposed to react to these changes?

The loss of a cluster node impacts the exchange of messages both to message senders and message receivers.

For message receivers, there is always the possibility that an expected message will never be received. The receiving actor needs to take this into consideration. There needs to be a plan B. The fact that expected messages may not be received is a fact of life with asynchronous messaging. It is also true that in most cases dealing with lost incoming messages does not require some level of cluster awareness.

Cluster Aware Actors

On the other hand, it is often necessary for message senders to have some level of cluster awareness. Router actors can handle the logistics of sending messages to other actors that may be distributed across the cluster. A router actor receives messages, but it does not handle the message itself. It forwards the message to a worker actor. These worker actors are often referred to as routees. The router actor is responsible for routing messages to other routee actors based on a routing algorithm. The actual routing algorithm used varies based on the specific requirements of each router. Examples of routing algorithms are round robin, random, smallest mailbox, etc. See the Java docs for the interface RoutingLogic for more examples of router implementation patterns.

cluster aware router

Figure 2 - Cluster aware router

Consider this example scenario as shown in Figure 2. Recall that the client that sends a message to an actor has no idea how that actor will handle the message. The receiving actor is a black box from the perspective of the client. The recipient actor may delegate the work to be done to other worker actors. The recipient actor, in this case, could be a router. It routes incoming messages to delegate routee actors that do the work.

In this example scenario, the router actor could be cluster aware, and it could be routing messages to actors that are distributed across nodes in the cluster. So what does it mean to be cluster aware?

Cluster aware actors use information about the composition of the current cluster state to make decisions about how route incoming messages to other actors that are distributed across the cluster. One of the most common use case for cluster aware actors is routers. Cluster aware routers decide how to route messages to routee actors based on the current state of the cluster. For example, a router that knows the location of routee actors that are distributed across the cluster routes messages to routees based on a distributed work algorithm.

Out of the box, Akka provides a nice collection of routers and all of these routers can be made to be cluster aware. The Akka documentation covers this in great detail. See the Routing and Cluster Aware Routers sections of the Akka documentation for more information on how to implement them.

Custom Cluster Aware Actors

While the Akka provided cluster aware routers are helpful in some cases you may need to implement custom routers that better fit your specific needs. Creating custom cluster aware routers is not that hard to do, and they can be a lot of fun to design and implement.

There are five main ingredients for implementing cluster aware routers. 1. An understanding of actor to actor collaboration 1. The mechanics of routers 1. How to subscribe to cluster events 1. How to access the topology of the cluster 1. How to react to nodes leaving and joining the cluster

Actor collaboration (point 1) involves actors working together. These actors exchange messages that pass information and to delegate tasks from one actor to another. With cluster aware routers a typical collaboration pattern is that a router actor resides on each node in the cluster or on each node with a particular role. These router actors handle routing messages to local routee actors and know how routee actors are distributed across the cluster.

The mechanics of the cluster aware routers (point 2) includes operations for sending messages to the right node. As an example, say you are implementing a collection of worker actors that are distributed across the cluster. Each router knows how to route incoming messages to specific workers.

router forwarding a message to a local routee

Figure 3 - Router forwarding a message to a local routee

If a router receives a message for a worker the is located on the same node as the router itself (as shown in the Figure 3) then the router simply forwards the message to the local worker.

message forwarded from one router to another

Figure 4 - Message forwarded from one router to another

On the other hand, if a router receives a message for a worker that is located on another node (as shown in the Figure 4) it forwards the message to the router that is resident on the node that contains the remote worker. Each router uses a worker distribution algorithm that identifies the node location of each worker.

To be cluster aware an actor, in this case, a router actor, needs to know when the topology of the cluster changes (point 3). This is done by registering to be notified of cluster changes via messages from the cluster. To be notified an actor must Subscribe to Cluster Events. For an example of how this is done see the example code in the Subscribe to Cluster Events section of the Akka documentation.

The cluster event messages do provide information about the current topology of the cluster (point 4). It is also possible to obtain this information directly via a call to the Cluster.get(system).state() method. This state() method returns a ClusterEvent.CurrentClusterState object that provides methods for accessing information about the cluster, such as a list of the current cluster member nodes.

When nodes leave or join the cluster (point 5), this may impact the distribution of routee actors on each node. As an example, say you have implemented a routee distribution algorithm that is based on consistent hashing, which is a common approach for distributing a collection across a dynamically changing set of nodes.

node leaving the cluster

Figure 5 - Node leaving the cluster

When a node leaves the cluster the routees that were located on that node will have to be redistributed to the remaining nodes in the cluster (see Figure 5 above). This will require that each router is aware of the node topology change and they must be capable of handling the routee redistribution.

The Foundation for Resilience and Elasticity

Akka clusters provide a solid foundation for building systems that are resilient to failure and also capable of elastically expanding and contracting as needed to handle the current processing load. Cluster aware actors are one the primary building blocks that enable you to build systems that react as needed when the topology of the cluster changes.

Your subscription to the Lightbend Reactive Platform includes enterprise capabilities that extend functionality beyond what’s available in the open source projects to deliver an enterprise-grade platform ready for your mission-critical systems. Learn more about how to use the Akka commercial features every month with our new Akka Tip Of The Month.

If you have general questions about how to utilize your subscription better, please contact support. If you have additional questions, or need assistance building an Akka cluster open a support ticket or email, and we will be happy to help.