Akka.net is a .NET port of the java based Akka toolkit and runtime for building distributed systems following the actor model. Akka.net is compatible with PCF but there some challenges that are purely technical and in how to automate the recovery of certain scenarios like split brain clusters.
Akka nodes in a cluster will require TCP connectivity between all nodes. On PCF an Akka node is an application instance. Provided the Application Security Groups (ASG) of the PCF foundation allow it and you know the public IP and Port of the app instance this should be possible.
It’s also possible technologies like NSX could prevent the communication. We’ve successfully tested akka nodes on Windows2012r2 and Linux cells.
Whilst most messaging with Akka.net usually occurs over TCP you’ll probably not need TCP routing unless applications outside of PCF will have to connect into the cluster. TCP routing might introduce more challenges as you’d potentially have multiple nodes behind a single address.
Akka has seed nodes that are designated as contact points for joining the cluster. All nodes (seed and non-seed) connect to these nodes and discover all nodes in the cluster. Nodes are then free to communicate directly.
When pushing akka nodes to Linux cells on PCF you must disable IPv6 and enable the enforce-ip-family flag within the Remote configuration section. Linux cells on PCF have the IPv6 interfaces disabled and this results in an error.
remote {
dot-netty.tcp {
transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
transport-protocol = tcp
dns-use-ipv6 = false
enforce-ip-family = true
}
}
Lighthouse is a sample Seed Node provided by PetaBridge, it supports both .NET and dotnet core.
The challenge with running Akka on PCF is first ensuring the Lighthouse nodes are UP prior to any non-seed nodes joining the cluster. Secondly as the address changes the non-seed needs to discover the seed node addresses when starting so they can join the cluster.
One simple approach is to use Redis as a basic discovery mechanism. Lighthouse nodes start and register their IP in Redis, non-seed nodes wait for an address to be present in Redis. Lighthouse nodes also use the list of seed node IPs as they need to connect to each other.
This repo contains a basic approach for doing this on PCF: https://github.com/miclip/AkkaCluster
On PCF a split cluster can occur when all seed nodes go down together. The platform will restart them but with different addresses. The non-seed nodes will continue but they’ll obviously lose connectivity with the seed nodes. When the seed nodes start they’ll form their own cluster. You’ll now have 2 separate clusters, one for all seed nodes and another for all non-seed nodes. Akka does come with a Split Brain Resolver to address some of these issues but this is still a possible scenario.