Akka.Remote: MessageContainerSerializer throws misleading error message when can't deserialize payload

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.4.20
Which Akka.NET Modules? Akka.Remote

Describe the bug
When deserializing an ActorSelection mesage that contains an unserializable payload, we throw an error message that looks like this:

[2021/06/02 16:56:59.792] [Error] [Akka.Actor.OneForOneStrategy] Exception has been thrown by the target of an invocation.
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.Runtime.Serialization.SerializationException: Cannot find serializer with id [13] (manifest [AA]). The most probable reason is that the configuration entry 'akka.actor.serializers' is not in sync between the two systems.
at Akka.Serialization.Serialization.Deser

This gets interpreted as "could not find serializer with Id=6", which is not true.

To Reproduce

Steps to reproduce the behavior:

1. Deserialize ActorSelectionMessage with unserializable payload (non-registered payload)
2. Log error message

Expected behavior
Should clearly state that it was the wrapped message for which deserialization failed, not the higher level MessageConatinerSerializer.

Environment
Windows and Linux

[Aaronontheweb]

More error details @Arkatufus

[2021/06/02 16:56:59.792] [Error] [Akka.Actor.OneForOneStrategy] Exception has been thrown by the target of an invocation.
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.Runtime.Serialization.SerializationException: Cannot find serializer with id [13] (manifest [AA]). The most probable reason is that the configuration entry 'akka.actor.serializers' is not in sync between the two systems.
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.DistributedData.Serialization.SerializationSupport.OtherMessageFromProto(OtherMessage other)
at Akka.DistributedData.Serialization.ReplicatedDataSerializer.GenericLWWRegisterFromProto[T](LWWRegister proto)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at Akka.DistributedData.Serialization.ReplicatedDataSerializer.LWWRegisterFromProto(LWWRegister proto)
at Akka.DistributedData.Serialization.ReplicatedDataSerializer.LWWRegisterFromBinary(Byte[] bytes)
at Akka.DistributedData.Serialization.ReplicatedDataSerializer.FromBinary(Byte[] bytes, String manifest)
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.DistributedData.Serialization.SerializationSupport.OtherMessageFromProto(OtherMessage other)
at Akka.DistributedData.Serialization.ReplicatorMessageSerializer.DataEnvelopeFromProto(DataEnvelope proto)
at Akka.DistributedData.Serialization.ReplicatorMessageSerializer.WriteFromBinary(Byte[] bytes)
at Akka.DistributedData.Serialization.ReplicatorMessageSerializer.FromBinary(Byte[] bytes, String manifest)
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.Remote.Serialization.WrappedPayloadSupport.PayloadFrom(Payload payload)
at Akka.Remote.Serialization.MessageContainerSerializer.FromBinary(Byte[] bytes, Type type)
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.Remote.MessageSerializer.Deserialize(ExtendedActorSystem system, Payload messageProtocol)
at Akka.Remote.DefaultMessageDispatcher.Dispatch(IInternalActorRef recipient, Address recipientAddress, Payload message, IActorRef senderOption)
at Akka.Remote.EndpointReader.<Reading>b__11_0(InboundPayload inbound)
at lambda_method32(Closure , Object , Action`1 , Action`1 , Action`1 )
at Akka.Actor.ReceiveActor.OnReceive(Object message)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location ---
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)