Support consuming and producing nuget packages with native binaries

[davidfowl]

We need a convention for both consuming and adding native binaries to nuget packages.

When you're developing a project and need to take a dependency on a native binary, you need to put it in a path that is loadable. This scenario needs to take bitness and OS into account.

You also need to be able to produce a package that has these native binaries inside of it.

[monoman]

Finally someone echoes my 3-years old issue: see
https://nuget.codeplex.com/workitem/1221

Rafael Teixeira
O..:.)oooo

On Tue, Jul 1, 2014 at 1:32 AM, David Fowler [email protected]
wrote:

We need a convention for both consuming and adding native binaries to
nuget packages.

When you're developing a project and need to take a dependency on a native
binary, you need to put it in a path that is loadable. This scenario needs
to take bitness and OS into account.

—
Reply to this email directly or view it on GitHub
#402.

[davidfowl]

Data points:

Here are some existing projects using native dependencies:

• Kestrel
• DataCommon.SQLite
• Helios (today helios has a native project in the same solution as the managed project)

Kestrel has binaries committed to source control in this format:

$/src/Microsoft.AspNet.Server.Kestrel/native/win/x86
$/src/Microsoft.AspNet.Server.Kestrel/native/win/amd64
$/src/Microsoft.AspNet.Server.Kestrel/native/darwin/universal

It also keeps the same structure in the nupkg itself:

https://github.com/aspnet/KestrelHttpServer/blob/dev/build/_custom-goals.shade#L12

The build script above puts the native binaries at the root of the nuget package with the same layout.

At runtime, it loads the native library using the following method:

https://github.com/aspnet/KestrelHttpServer/blob/e4b9bd265c75704529409638fd9cdfac504a93ef/src/Microsoft.AspNet.Server.Kestrel/KestrelEngine.cs#L26

https://github.com/aspnet/KestrelHttpServer/blob/dev/src/Microsoft.AspNet.Server.Kestrel/Networking/Libuv.cs#L23

Ideally, with the KRuntime, we should have conventions around what directories we automatically add for loading native binaries (we the option of specifying this in the project.json). Of course the library author can always manually load library, but that only really works if you can package all native dependencies in the nuget package. This doesn't work well on *nix, see aspnet/KestrelHttpServer#10 for more details.

SqlLite is a bit more interesting:

The build process downloads sqlite dynamically and for packaging, it uses an msbuild script on net451 so that nuget will automagically copy the binaries into the bin folder of the project.

https://github.com/aspnet/DataCommon.SQLite/blob/dev/makefile.shade#L46
https://github.com/aspnet/DataCommon.SQLite/blob/dev/src/Microsoft.Data.SQLite/build/net451/Microsoft.Data.SQLite.targets

At runtime it uses load library (similar to kestrel):

https://github.com/aspnet/DataCommon.SQLite/blob/dev/src/Microsoft.Data.SQLite/Utilities/NativeLibraryLoader.cs#L16

[davidfowl]

@lodejard and @GrabYourPitchforks Need your POV on this. Assuming we invent a convention for determining where native assets get loaded from within a package based on OS and Bitness, we'll either modify PATH or LD_LIBRARY_PATH.

[GrabYourPitchforks]

Thou shalt not modify the environment block (including environment variables) while the application is running. It's not yours to mess with.

[davidfowl]

Another option might be to call LoadLibrary (and dlopen on *nix) on the appropriate binary. The only downside to this might be that we would load the library eagerly instead of lazily.

[lilith]

Another project that uses native dependencies is Microsoft.SqlServer.Compact, which @davideboo says works pretty well. The scripting in that package is a bit hairy.

Moving that responsibility off of package developers and onto tooling would be great - as it's all tooling specific, and if an interface changes, old packages will become unusable.

I've been collecting thoughts on github here; hopefully I'll be able to reorganize them as a set of best practices after some more thorough testing of the various assumptions and hypothesis.

[davidfowl]

We're moving to a model where packages are declarative and client can do the right thing for the platform.

[lilith]

Is there any documentation of this model? If I submit a PR to paket that enables this workflow, I'd like to keep the nuget syntax compatible or consistent with what you guys are doing.

[jbrwn]

Is there any more information on this? Seems like there are a few issues out there about this but no resolution.

#990
NuGet/Home#300

An update would be much appreciated.

[jbrwn]

pinging @davidfowl and @yishaigalatzer

Would love to get a quick status report on this topic.

My use case is for a c++/cli project i maintain: https://github.com/jbrwn/NET-Mapnik. My current nuget strategy is to copy all dependant native dlls to the output dir using a .props file in the build directory of my nuget package. This model falls apart with dnx and would love to find an alternative solution.

[yishaigalatzer]

We should have something out within a week or so. Still working on documenting the details

[yishaigalatzer]

I will try to post a sample today or tomorrow

[yishaigalatzer]

https://www.myget.org/F/dotnet-core/api/v2/package/System.IO.Compression/4.0.0-beta-22927

[jbrwn]

@yishaigalatzer - thanks for the response and posting the sample package. I'll be keeping my eye out for documentation. In the mean time, is it possible for you to elaborate a little on how this all works? I see an "any" TFM directory and a runtime.json file in the root of the package. It would be great to hear about how this all comes together so i can start experimenting with it even if complete documentation is not ready.

[yishaigalatzer]

The any folder is going to probably be renamed before we are done. The runtimes is about bringing implementation that are runtime target appropriate. We can probably post more preliminary info here to get you going, let me see if we have something useful

[glennc]

We are talking about a package that has a managed assembly that needs to invoke stuff from a native assembly that it carries, right? If that's the case then the current plan, as far as I understand it, is to be able to create a package like the following:

MyLib
  lib
    dotnet
      `MyLibrary.dll`
  runtimes
    win-x86
        native
            `MyNativeLib.dll`
    win-x64
        native
            `MyNativeLib.dll`

The build system responsible for consuming this package would need to know to copy the native assembly to your build output when building for the given runtime. You could put linux-x86, linux-64, etc as well. The key is that there is a runtimes folder with a runtime id. The runtime Id is how you pivot on things like os and architecture. The native folder is for assets that need to be copied to the applications output directory to be consumed by my library.

@davidfowl or @anurse might have more detailed comments. There should be more docs on how it all fits together coming at some point soon.

[lilith]

@glennc Any news on which build systems will be supporting this? Will native package support be exclusive to VS2015 and dnx-based systems?

Also, have there been corresponding changes in the .NET full framework to allow native libraries to be located even when the managed libraries are loaded with shadow-copying?

[jbrwn]

@glennc Thanks for your input.

We are talking about a package that has a managed assembly that needs to invoke stuff from a native assembly that it carries, right?

That's right.

The build system responsible for consuming this package would need to know to copy the native assembly to your build output when building for the given runtime.

I think this is the primary question for me and others - how will dnx/dnu/visual studio handle native assemblies in a nuget package?

In the past we could always just make up our own convention for including native dlls in a nuget package and then copy them wherever we want using a Install.ps1/init.ps1 script or a .props file. However, the game is changing because, as far as i know, there is no "output directory" in dnx - everything is loaded into memory from the dependent nuget packages.

[natemcmaster]

@moozzyk are their issues/PRs on the dotnet repos we can follow to track this?

[moozzyk]

@natemcmaster - I think the change on the CoreClr side went in last week so is no longer blocking.

[natemcmaster]

@moozzyk awesome.

[lilith]

Can someone writeup the status of native binary packages across all 2015 platforms (full .NET framework, VS 2105, VS code, DNX, etc)?

[moozzyk]

@nathanaeljones - the support in DNX is scheduled for RC. Can you elaborate more on what your expectations are with regards to "VS 2015 and VS Code" ?

[lilith]

• Is there documentation or tooling to guide creation of valid native packages?
• Can a VS2015 user install a native package into a classic .NET 4.6 full project, and have unit tests 'just work', I.e, test runners will locate the appropriate native binary at runtime?
• Does install and test work under DNX environments?
• Will native packages ever work on VS2013?

[moozzyk]

@nathanaeljones - I see. No. This bug is only about being able to consume packages containing native dlls in dnx - i.e. we need to make sure that if you have a package that depends on a package that contains native binaries you can just use the DllImportAttribute and dnx is able to find and load the native library so that you don't have to do it manually. This also should work on other platforms - Linux and Mac OS

Side note - it is possible to create nuget packages containing native binaries even today (examples:cpprestsdk, SignalR C++ client, some documentation). I have not tried creating a NuGet package containing both native and managed dlls nor have I tried creating a package that contains a native dll that is being consumed by managed code but I think you can make it work just by adding a build step that copies the native dll to the same folder where the managed dll is copied. I believe CLR (LoadLibrary invoked by CLR) will try searching for the native dll in the folder where the managed dll with the DllImportAttributes is so things should work. This should also work in VS2013. However, the packages with native dlls we will use in dnx will have a different structure - I am not sure if it has been documented yet. As always you can create a package just be creating the right structure in a folder and runing the nuget pack command. Again, I assume that you can just add a build step to copy the native lib to the same folder as the managed lib to consume this dll by a manged assembly in non-dnx projects.

[natemcmaster]

@moozzyk Any updates on this?

[moozzyk]

@natemcmaster - #3049

[lilith]

Build steps do not work for classic ASP.NET or projects with unit tests (which should be all of them!), as those assemblies get left behind by the shadow copier.

[moozzyk]

Fixed in b9f0af9 and cba17e0.
Related issue #3073

[oryol]

Finally, I didn't understand what should I do to make it works? For example I reference Gtk# package (https://www.nuget.org/packages/GtkSharp.Win32) and target dnx46. When I run project Gtk.Application.Init complains that native library wasn't loaded. May be this package hasn't upgraded to the some new conventions for DNX (as I see it has only old plain MSBuild steps to copy native dll-s). I can update it myself but I need to understand how it can be done?

[moozzyk]

@oryol - this might be useful for you (though things are changing a bit in RC2 with dnx being replaced by dotnet)

[davidfowl]

That will still work in RC2.

[rmarskell]

@davidfowl Does this still work? I'm trying to create a wrapper library (netstandard1.6) for a C++ dll from another company. I've put the dll files in the appropriate folders ({project root}/runtimes/win-x64/native/mylib.dll) and then attempted to reference methods in the library using [DllImport( "mylib" )] but it doesn't seem to be finding it. It works if I specify the full path instead of just "mylib", but that's obviously not helpful. Not sure if I'm missing something here or if it has been changed.

[davidfowl]

No this doesn't work. You need to put it in a package. Or copy it to the output folder on build

[rmarskell]

Oh, that's a shame, but at least it explains why I couldn't get it to work. Forgive my ignorance, but is there an optimal way to make a post-publish copy script that will have access to the publish architecture (x86/x64) and platform (windows/linux/etc.)?

[paolo8417]

You need to put it in a package

@davidfowl , can you explain how ?!

Thanks

[moozzyk]

@paolo8417 - take a look at the post I wrote some time ago. This was written before RC1 shipped and some things no longer work the same way but it should give you the general idea. Then look at how we build the libuv package: https://github.com/aspnet/libuv-package