TypeLoadException in UDF

[quarkonium3]

I'm trying to work up a project calling into some of our existing .net code using a UDF. At the moment I'm just using the example spark-submit on single windows 10 machine (spark 3.0.1):

spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-3-0_2.12-1.0.0.jar dotnet SparkExec.dll

Simpler examples work as expected, however trying to call the UDF fails with the following:
---> System.TypeLoadException: Could not load type 'System.IO.Path' from assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

This doesn't make a great deal of sense to me. Normally I think of this as meaning it doesn't know where to load the type from the assembly, but it it can successfully call the same assembly outside the UDF. I'm at a bit of a loss how to diagnose further.

Any help appreciated

[Niharikadutta]

Hi @quarkonium3 Thanks for your question! Have you set DOTNET_ASSEMBLY_SEARCH_PATHS environment variable to where your assemblies are stored? The worker looks at this path to load the assemblies it needs in order to execute the UDF. Please take a look at this doc for more details on this.

[Niharikadutta]

Are you using .NET framework or .NET Core to build your application?

[quarkonium3]

.NET Core - actually I migrated it to .NET 5, so I suppose I don't say 'Core' anymore

[imback82]

@quarkonium3 Do you have a simple repro app we can try?

[quarkonium3]

@imback82 I made a repro app today. How can I get it to you?
--> Invited you to a repo

[Niharikadutta]

Could you send it to this email id - [email protected] ?

[quarkonium3]

Ok sent

…

On Thu, Feb 4, 2021 at 11:27 PM Niharika Dutta ***@***.***> wrote: Could you send it to this email id - ***@***.*** ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#817 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKN7NQU7MTC23SGQCLYNM73S5KHEPANCNFSM4W4IIWDQ> .

[Niharikadutta]

Thanks @quarkonium3 , I ran your application and was able to reproduce the issue. However, it works with Spark 2.4, could you please try to see if Spark 2.4 also works for you with your original app? Meanwhile, I will investigate further and get back to you.

[quarkonium3]

I tried 2.4 and initially thought it works. However, turns out there is something different about the way the csv reader interprets NULL.
I modified the lines that read the csv and now get the same behaviour in the minimal repro (just need to add .Option("nullValue", "NULL")

So the two csv reader lines are now:
var locs = spark.Read().Schema(locSchema).Option("header", true).Option("nullValue", "NULL").Csv("loc_test.csv");
var covs = spark.Read().Schema(covSchema).Option("header", true).Option("nullValue", "NULL").Csv("cov_test.csv");

With that change the minimal app fails the same way. I've modified in the repo and invited you as well (or I can send you an updated version if needed)

[quarkonium3]

Alright! I converted the minimal app to compile in .net core 3.1 and it works under both spark 2.4.7 and 3.0.1
(I had to fix up some of the data conversions in Row.GetAs (decimal and Enums not allowed it seems) and it doesn't like some Json deserialisation so commented that out - I never got that far under .net5 so wasn't aware of the problem. You'll see it fail a different way if you try it).
I pushed these changes to the minimal app repo if you're interested (happy to send along via email if you want)
Anyway - it seems the main issue is with .net5

[imback82]

Anyway - it seems the main issue is with .net5

Did you try compiling worker binary with .NET5 and see if it helps?

spark/src/csharp/Microsoft.Spark.Worker/Microsoft.Spark.Worker.csproj

Line 5 in d608d8e

<TargetFrameworks>net461;netcoreapp3.1</TargetFrameworks>

[quarkonium3]

Aha! Ok that works. Thanks for the tip
You have to kill off the compile-time 'BinaryFormatter serialization is obsolete' with
<NoWarn>$(NoWarn);SYSLIB0011</NoWarn>
But I'm sure you know about that :)