DData: Cannot create a shard proxy on a cluster node that is not in the same role as the proxied shard entity

[ondrejpialek]

Running Akka 1.3.5 inside a netcoreapp2.0 hosted on Service Fabric.

I have the following setup:

1. Backend service

   • Hocon:

   cluster {
   	roles = [backend]
   
   	sharding {
   		role = "backend"
   	}
   }
   

   • Shard:

   ClusterSharding.Get(ActorSystem).Start(
   	typeName: Entity.Product,
   	entityProps: Props.Create(() => new Product()),
   	settings: ClusterShardingSettings.Create(ActorSystem).WithRole(ClusterRole.Backend),
   	messageExtractor: new MessageExtractor());
   

2. Api service

   • Hocon:

   cluster {
   	roles = [api]
   
   	sharding {
   		role = "api"
   	}
   }
   

   • Shard:

   var products = ClusterSharding.Get(ActorSystem).StartProxy(
   	typeName: Entity.Product,
   	role: ClusterRole.Backend,
   	messageExtractor: new MessageExtractor());
   

Works fine with Persistance backing cluster sharding but if I switch to ddata I get the following error:

Error while creating actor instance of type Akka.DistributedData.Replicator with 1 args: (Akka.DistributedData.ReplicatorSettings) ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: The cluster node akka.tcp://my-system@localhost:30020 does not have the role backend

Seems like this line:

akka.net/src/contrib/cluster/Akka.DistributedData/Replicator.cs

Line 353 in c954b79

if (!string.IsNullOrEmpty(_settings.Role) && !_cluster.SelfRoles.Contains(_settings.Role))

requires that the node that creates the proxy to also be in the same role of the entities it wants to communicate with. In my example the Api service is only in the api cluster role, not backend role, hence the exception. As mentioned before with Persistence backing sharding there is no exception and I can communicate with my entities well via the Proxy.

Setting the ddata cluster role has no effect as the replicator's role is set during the creation of the shard proxy based on the cluster setting here:

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/ClusterShardingGuardian.cs

Line 262 in c954b79

var replicatorRef = Context.ActorOf(DistributedData.Replicator.Props(_replicatorSettings.WithRole(settings.Role)), name);

So it seems that I would need to set the cluster role of the Api service to backend if I wanted to communicate with entities deployed to the backend nodes. But in doing so I would allow some entities to be deployed to the ApiService, would I not? Maybe not if I only start the proxy here and not the full shard, but still seems a bit odd to need to set the backend role on the API service. Is this really needed?

Thank you,
Ondrej.

[ondrejpialek]

I found a quote from @Horusiath on #3277:

When specifying settings on start of cluster sharding region, you may specify the role, that all nodes containing regions of that type, should have:

var sharding = ClusterSharding.Get(system);
var region = await sharding.StartAsync(
   typeName: nameof(MyActor),
   entityProps: Props.Create<MyActor>(),
   settings: ClusterShardingSettings.Create(system).WithRole(nameof(MyActor) + "-region"),
   messageExtractor: new MessageExtractor());

All nodes with that role are expected to have shard regions started on them.

Regarding shard region proxy, here role also should be provided, but node, which hosts proxy shouldn't contain that role.

So the example I gave above is legit. I only managed to get DData running with a Proxy Shard when both nodes were in the same role ( [backend, sharding], [api, sharding] ) but as @Horusiath explained above this should not be set up that way and indeed I started getting errors occasionally:

Message Register from akka.tcp://my-system@localhost:30006/system/sharding/product to akka://my-system/system/sharding/productCoordinator/singleton/coordinator was not delivered. 3 dead letters encountered.

The node with the real shard (not proxy) could not start the coordinator as it was not Oldest:

ClusterSingletonManager state change [Start -> Younger] Akka.Cluster.Tools.Singleton.Uninitialized

So I guess I will have to roll-back to Persistence once again as DData does not seem to work well Shard Proxies at the moment.

[ondrejpialek]

I might be way off, but in Akka the replicator for a proxy is set to dead letters, it does not create a replicator for the proxy at all....
https://github.com/manonthegithub/akka/blob/3639226b56a8d1978c0fd1b94c4502d57f63b1fc/akka-cluster-sharding/src/main/scala/akka/cluster/sharding/ClusterSharding.scala#L668

[ondrejpialek]

I tested a new version of the code locally (see PR) in the scenario described above and I no longer received those errors and the shard proxies were working correctly.