[BUG]: ForeachWriter does not work in net framework

[nbalexiev]

ForeachWriter fails when executed in .NET Framework. An easy repro is the E2E "TestForeach" executed with the net framework worker.

Steps to reproduce the behavior:

1. Get my fork of the repo here: https://github.com/nbalexiev/Spark.NetFork
2. Execute: DataStreamWriterTests.TestForeach from Microsoft.Spark.E2ETest.Framework
3. It will fail inside UdfSerDe.DeserializeTargetData with:

System.ArgumentException: Object of type 'Microsoft.Spark.E2ETest.IpcTests.DataStreamWriterTests+TestForeachWriter' cannot be converted to type 'Microsoft.Spark.Sql.IForeachWriter'.
at System.RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
at System.Reflection.RtFieldInfo.InternalSetValue(Object obj, Object value, BindingFlags invokeAttr, Binder binder, CultureInfo culture, StackCrawlMark& stackMark)
at System.Reflection.RtFieldInfo.SetValue(Object obj, Object value, BindingFlags invokeAttr, Binder binder, CultureInfo culture)
at System.Reflection.FieldInfo.SetValue(Object obj, Object value)
at Microsoft.Spark.Utils.UdfSerDe.DeserializeTargetData(TargetData targetData) in C:\github\dotnet-spark\src\csharp\Microsoft.Spark\Utils\UdfSerDe.cs:line 237
at Microsoft.Spark.Utils.UdfSerDe.Deserialize(UdfData udfData) in C:\github\dotnet-spark\src\csharp\Microsoft.Spark\Utils\UdfSerDe.cs:line 179
at Microsoft.Spark.Utils.CommandSerDe.DeserializeUdfs[T](UdfWrapperData data, Int32& nodeIndex, Int32& udfIndex) in C:\github\dotnet-spark\src\csharp\Microsoft.Spark\Utils\CommandSerDe.cs:line 335
at Microsoft.Spark.Utils.CommandSerDe.Deserialize[T](Stream stream, SerializedMode& serializerMode, SerializedMode& deserializerMode, String& runMode) in C:\github\dotnet-spark\src\csharp\Microsoft.Spark\Utils\CommandSerDe.cs:line 313
at Microsoft.Spark.Worker.Processor.CommandProcessor.ReadRDDCommand(Stream stream)
at Microsoft.Spark.Worker.Processor.CommandProcessor.Process(Stream stream)
at Microsoft.Spark.Worker.Processor.PayloadProcessor.Process(Stream stream)
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete)

• OS: Windows

NOTE: This works fine in .Net Core. The exact same test passes without any issues.
I've discussed the issue with @suhsteve

[suhsteve]

@nbalexiev can you post the exception message as well ?

[nbalexiev]

@suhsteve Added inline in the initial comment.

[nbalexiev]

It repros in net 472 as well.

Here's a hacky way to do a dynamic cast using LINQ expression:
ParameterExpression paramExpression = Expression.Parameter(typeof(object), "value");
BlockExpression castBlockExpression = Expression.Block(Expression.Convert(Expression.Convert(paramExpression, field.Value.GetType()), targetType.GetField(field.Name, BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic).FieldType));
Expression.Lambda(castBlockExpression, paramExpression).Compile().DynamicInvoke(field.Value)

This results in:

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Delegate.DynamicInvokeImpl(Object[] args)

innerException: {"Unable to cast object of type 'TestForeachWriter' to type 'Microsoft.Spark.Sql.IForeachWriter'."}

At the same time the following checks are true:

• field.Value is Microsoft.Spark.Sql.IForeachWriter -> true
• typeof(Microsoft.Spark.Sql.IForeachWriter).IsAssignableFrom(field.Value.GetType()) -> true