During CoordinatedShutdown an exception 'Hyperion.ValueSerializers.UnsupportedTypeException: No coercion operator is defined between types 'Interop+Sys+SocketEvent*' and 'System.Object'.' is thrown

[wesselkranenborg]

I got a similar issue as mentioned in #4860. I tested this with the nightly version of 1.4.19 as there #4860 is fixed.

<PackageReference Include="Akka" Version="1.4.19-beta637539553808642405" />
<PackageReference Include="Akka.Cluster" Version="1.4.19-beta637539553808642405" />
<PackageReference Include="Akka.Cluster.Sharding" Version="1.4.19-beta637539553808642405" />
<PackageReference Include="Akka.Cluster.Tools" Version="1.4.19-beta637539553808642405" />
<PackageReference Include="Akka.DependencyInjection" Version="1.4.19-beta637539553808642405" />
<PackageReference Include="Akka.Persistence.Query.Sql" Version="1.4.19-beta637539553808642405" />
<PackageReference Include="Akka.Persistence.SqlServer" Version="1.4.16" />
<PackageReference Include="Akka.Serialization.Hyperion" Version="1.4.19-beta637539553808642405" />

But am still getting this error on shutdown of my ActorSystem. @Aaronontheweb mentioned that it was related to #4860 but it looks not related.

The error which is thrown is:

[ERROR][04/14/2021 19:38:05][Thread 0082][[akka://blueprint-service/system/IO-TCP#1876155147]] Swallowing exception during message send
Cause: Hyperion.ValueSerializers.UnsupportedTypeException: No coercion operator is defined between types 'Interop+Sys+SocketEvent*' and 'System.Object'.
   at Hyperion.ValueSerializers.UnsupportedTypeSerializer.WriteManifest(Stream stream, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method(Closure , Stream , Object , SerializerSession )
   at Hyperion.ValueSerializers.ObjectSerializer.<.ctor>b__9_1(Stream stream, Object o, SerializerSession session)
   at Akka.Serialization.HyperionSerializer.ToBinary(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserializePayload(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserialize(Envelope envelope)
   at Akka.Actor.ActorCell.SendMessage(Envelope message)

This is thrown when a CoordinatedShutdown is initiated from the ActorSystem. I can also reproduce this in 1.4.18 (not sure about 1.4.17 but in our stable branch we use 1.4.16 and there it doesn't happen).

[wesselkranenborg]

Maybe good to additionally mention that we have the following settings enabled in our HOCON:

akka.actor.serialize-creators = on
akka.actor.serialize-messages = on

[Aaronontheweb]

Maybe good to additionally mention that we have the following settings enabled in our HOCON:

akka.actor.serialize-creators = on
akka.actor.serialize-messages = on

I think this is the real bug - we had an issue with our testkit that effectively hid these serialization errors from unit tests prior to Akka.NET v1.4.17. I ran into this in Petabridge.Cmd's own test suite myself.

The fact that Akka.Actor.ActorCell.SerializeAndDeserializePayload is being called - this only occurs when serialize-messages = on and in this case it looks like Hyperion is trying to serialize a SocketEvent, which is impossible to serialize. So we probably have some built-in Akka.IO messages that need to be marked with INoSerializationVerificationNeeded

[Arkatufus]

@wesselkranenborg Are you, by any chance, using UDP?

[wesselkranenborg]

@wesselkranenborg Are you, by any chance, using UDP?

@Arkatufus We are not explicitly using UDP. But we are using Phobos, don't know if that is internally using UDP. These extra parameters are logged in the logging so that's why I am mentioning this.

| ActorPath | akka://blueprint-service/deadLetters
  | CorrelationId | None
  | LogSource | [akka://blueprint-service/system/IO-TCP#1705556657]
  | Message | Swallowing exception during message send
  | MessageTemplate | {Message:l}
  | ServiceName | blueprint-service
  | SourceContext | Phobos.Actor.PhobosActorCell
  | Thread | 0048
  | Timestamp | 2021-04-22T06:27:42.0675268Z

[wesselkranenborg]

I see a kind of similar issue now when trying to connect using pbm. This also throws an exception in Akka.IO.TcpListener. So don't know if it's an issue with the TcpListener or with the PBM messages (or Phobos in the previous example) which are not serializable.

[09:36:56.892 +02:00 Error] [Akka.IO.TcpListener] [RequestId: ] Swallowing exception during message send
Hyperion.ValueSerializers.UnsupportedTypeException: No coercion operator is defined between types 'System.Threading.NativeOverlapped*' and 'System.Object'.
   at Hyperion.ValueSerializers.UnsupportedTypeSerializer.WriteManifest(Stream stream, SerializerSession session)
   at Hyperion.Serializer.Serialize(Object obj, Stream stream)
   at Akka.Serialization.HyperionSerializer.ToBinary(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserializePayload(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserialize(Envelope envelope)
   at Akka.Actor.ActorCell.SendMessage(Envelope message)

[Aaronontheweb]

Petabridge.Cmd uses Akka.IO.Tcp

[Swoorup]

So it appears that I can still reproduce this exact issue.