Akka Clustering and Remoting

Akka Tip of the Month - May 2017

By Hugh McKee, May 19, 2017

While there is a wealth of documentation around Akka, based on our real-world experience with users and customers over the last few years, we realize that there is a need for tutorial-style tips that focuses on different aspects of Akka. This led us to launch what we think is a highly-valuable “Akka Tip of The Month” series. In this first installment we’re going to focus on Akka Clustering vs. Akka Remoting.

At its core, Akka is often used for at least one of three reasons. First, Akka provides a powerful yet simple solution for handling concurrency with actors. Second, Akka Streams is a popular and relatively new feature that provides a functional stream based form of pipeline processing with backpressure. A third reason for using Akka is that with Akka it is possible to create systems that span multiple JVMs running across a distributed network. In effect, with Akka, you can create clustered virtual machines. You are no longer constrained to run on a single system in a single JVM.

Of course, concurrency, streams, and clustering are not the only reasons for using Akka. Akka is a toolkit, not a framework, and for a good reason. The Akka toolkit also provides many other capabilities, such as Event Sourcing and CQRS with Persistence, Persistence Query, and Cluster Sharding; Akka HTTP, Distributed Data, Alpakka, Configuration Checker and Diagnostics Recorder, Split Brain Resolver, and Akka Monitoring and Telemetry to name a few.

In this first Akka Tip of the Month, we will focus on how Akka is used to create distributed systems or clustered virtual machines.

From Brittle to Unbreakable

The clustering capabilities of Akka can be challenging to grasp at first. For many of us that have grown up writing code that, for the most part, runs in a series of synchronous steps in a single process and a single JVM, the concept of writing code that interacts asynchronously and that spans multiple JVMs on distributed nodes enters the realm of disbelief and magic. Be assured that the payoff for understanding how to use Akka clusters effectively is well worth the effort. Ask anyone that “gets it.”

In the Akka documentation, you will find the bulk of the material related to clustering in the Networking section. There is a lot of material in this part of the documentation. Much of the content dives into the implementation details, which is what documentation should do, but it can be somewhat overwhelming at first. So how do you avoid “not seeing the forest for the trees” or to and comprehend the big picture? Follow this Akka Tip of the Month where we will be providing monthly tips on all things Akka. Also, there is a lot of content out there about Akka and Akka clustering such as the Akka Blog: Let it crash, Stack Overflow, and many other sources such as the Lightbend Tech Hub, Get started with Lightbend’s technologies, and the Lightbend Tech Hub Blog.

In all of the material related to Akka clustering, including the Akka documentation, you will come across content in four general areas. The four areas are “real” Akka clustering, Akka Remoting, Akka Cluster Client, and finally other cluster related offerings that are various implementations built on top of clustering, for example, Cluster Sharding, Distributed Data, Cluster Singleton.

Akka Clustering

So what is “real” clustering? It is a group of JVMs, each running an actor system, and each configured as a cluster provider. The clue here is that the Akka configuration setting “akka.actor.provider” is set to “cluster”.

akka {
  actor {
    provider = "cluster"

Each actor system runs in a JVM. Multiple actor systems may be running in a single JVM, but the most common scenario is that there is a single actor system per JVM.

image alt text

Figure 1 - VM running Actor System in a JVM.

When a group of JVMs is running in this way, they form themselves into a tight-knit group of cluster members. These cluster members are equals; there is no master, no one member is in charge. The cluster members also closely monitor each other, they are constantly gossiping with each other in the background, watching out for new members that join the cluster and existing members that leave the cluster or are no longer reachable over the network.

This “real” cluster sets the stage for a clustered virtual machine. Actors running in this environment can just as easily receive messages that are sent from one member node to another as messages sent within a single JVM. In fact, there is no difference in the Java or Scala code for sending messages locally or for sending a message across the network.

someActorRef.tell(new SomeMessage(someMessageText), getSelf());

Of course, messages sent across the network takes more time to transmit and these messages are taking a more precarious route but it is still fast. The Akka team has put a considerable amount of effort into optimizing Akka clustering.

image alt text

Figure 2 - Akka Cluster.

An Akka cluster is an excellent foundation for building reactive systems. Akka clusters address all four of the fundamental properties of reactive systems. By design they are responsive, resilient, elastic and message driven. For a system to always be responsive, it must be both resilient and elastic. It is possible to build highly resilient systems with Akka because it is possible to build systems that can self-heal when member nodes leave the cluster due to failures. It is also possible to build systems that can be easily scaled up and down to handle peaks and valleys in system activity. The nice thing is that when you build a system with Akka clusters that is resilient, you get elasticity as well. The inverse is also true, build a system that is elastic, and it will also be resilient. The solution for resilience and elasticity with Akka clusters is to build systems that gracefully handle member nodes joining and leaving the cluster at any time. Finally, messaging is the lingua franca of Akka. The only way to communicate with actors is via asynchronous messages.

The real fun with Akka is designing and building systems with actors that run in a cluster. And the real power comes from creating actors that are cluster-aware. That is actors that are capable of reacting to changes in the topology of the cluster. These cluster-aware actors can do things like restore other actors that were formally running on a node that has left the cluster due to a failure or perhaps due to simply scaling down the number of nodes in the cluster. These cluster-aware actors can also react when new nodes join the cluster.

There are a number of cluster-aware solutions that are provided with the Akka toolkit. It is also easy for you to create custom cluster-aware actors. This is not your old-fashioned way for building systems. It is a very different way of thinking of system design and implementation, but when you start to wrap your head around these concepts, it is a very fun and interesting and very powerful way to do things.

Akka Remoting

Akka clustering is built on top of Akka remoting. Akka remoting provides some of the more basic features required by Akka clustering. With Akka remoting actors running in separate JVMs can communicate with each other. The primary difference between Akka clustering and Akka remoting is that with Akka remoting the individual actor systems do not form themselves into clusters. Each actor system runs independently of other remote instances.

image alt text

Figure 3 - Akka Remoting.

Each remoting instance also is configured with the property “akka.actor.provider” set to “remote.”

akka {
  actor {
    provider = remote

Sending messages to remote actors is simple.

ActorSelection selection =

For example, you can obtain an actor selection using the full actor location.

selection.tell("Pretty awesome feature", getSelf());

Then send the selected actor a message.

Akka remoting predates Akka clustering. While it is still an important part of the Akka toolkit, it is now mainly used internally by Akka itself. It is no longer intended to be used externally.

Akka remoting continues to be supported and documented mainly because of its legacy of use in the past. The documentation is also provided as a source of information for those that are curious about some of the internal workings of Akka.

Akka Cluster Client

With Akka cluster client it is possible to send messages to actors running in other clusters. This allows for a form of actor messaging between individual clusters.

While this form of messaging to actors in other clusters is supported, the recommended approach is to use Akka HTTP instead.

image alt text

Figure 4 - Akka Cluster Client.

Accessing actors on another cluster is handled differently. As shown in Figure 4, in each cluster actor system start a ClusterClientReceptionist. In the client actor system start a ClusterClient actor. Register the actors that may receive messages from cluster clients with the ClusterClientReceptionist. Message are sent via the ClusterClient. Here is an example.

ActorRef serviceA = system.actorOf(Props.create(Service.class), "serviceA");
ActorRef serviceB = system.actorOf(Props.create(Service.class), "serviceB");

First, in the cluster system create a few actors and register them with the ClusterClientReceptionist.

ActorRef c = system.actorOf(ClusterClient.props(ClusterClientSettings.create(system).withInitialContacts(initialContacts())), "client");

c.tell(new ClusterClient.Send("/user/serviceA", "hello", true), ActorRef.noSender());
c.tell(new ClusterClient.SendToAll("/user/serviceB", "hi"), ActorRef.noSender());

On the client system obtain an actor reference to the ClientCluster actor and use it as a gateway for sending messages. Please see the documentation for more details.

Other Cluster Related Offerings

The Akka team has a selection offerings that are built on the basic clustering capabilities of Akka. For the most part, these offerings provide solutions for things such as Event Sourcing, Command Query Responsibility Segregation (CQRS), and Conflict Free Replicated Data Types (CRDT) based distributed data. These solutions are implemented using actors running in a cluster. These same basic capabilities are available for you to use to create your solutions and, of course, you can expand the capabilities the come with the Akka toolkit.

Learn How To Use Clusters

You should also feel empowered to flex your clustering skills. While Akka Remoting and Cluster Client are interesting, with Akka clusters you can build systems that are resilient, elastic, and highly concurrent. For external access to Akka clusters use Akka HTTP or Akka Streams or Alpakka.

The Akka toolkit gives you the power to escape from the world of brittle single process systems into the world of unbreakable clustered virtual machines. Give it a try.

Your subscription to the Lightbend Reactive Platform includes enterprise capabilities that extend functionality beyond what’s available in the open source projects to deliver an enterprise-grade platform ready for your mission-critical systems. Learn more about how to use the Akka commercial features every month with our new Akka Tip Of The Month.

If you have general questions about how to utilize your subscription better, please contact support. If you have additional questions, or need assistance building an Akka cluster open a support ticket or email support@lightbend.com, and we will be happy to help.