Resolution of native libraries when using DllImport

[antoinebj]

TL;DR: How can I make DllImport automatically load the right native library from the runtimes/<rid>/native path when it is in a project referred by ProjectReference, similar to what PackageReference enables by adding the appropriate resolution information in deps.json?

--

If I create a NuGet package and put native DLLs in runtimes/<rid>/native/, the deps.json file generated for the assembly referencing that package will get runtimeTargets entries for the native libraries, allowing DllImport to find the appropriate DLLs to load. That mechanism is very useful for easily targeting multiple platforms with "Any CPU" despite reliance on native libraries that are each specific to a platform/architecture combination.

Now I would like to unit-test the classes in that package. Usually I put the unit test project in the same solution, make it refer to the project containing the classes to test. Unfortunately, that doesn't bring the mechanisms from PackageReference, as it only uses a ProjectReference. Therefore the deps.json for the unit test assembly doesn't get hints for the native libraries, and when the unit test is run, it complains about not finding the native DLLs.

A workaround would be to have a separate NuGet package containg nothing more than the native libraries, making the project containing .NET code refer to it, similar to what libgit2sharp does. However that's not convenient when updating the native libraries because it requires publishing their package before being able to test their integration in the .NET code.
I know about the NativeLibrary class where one can register a custom resolver, but that seems tedious for true multiplatform. I didn't find a simple way to replicate the RID-based resolution.

Is there any other way? Any thoughts?

[antoinebj]

I also noticed the existence of --additional-deps, but I don't know how to make unit test runners use that parameter.

[benvillalobos]

/cc: @vitek-karas

[vitek-karas]

I can only comment on the runtime side of things:

• If the info is not in .deps.json runtime won't find it - so it's up to SDK to put it there. As you noted this works for PackageReference. I don't know if or how it should for ProjectReference.
• --additional-deps won't help that much, it lets you specify additional .deps.json, but that assumes you have one. So you would have to "hand-craft" one.

[antoinebj]

This has been labeled as a question, which it is indeed, but only because I wouldn't want to make a feature request if there is already an existing solution.
If the conclusion to this question is that there is no simple solution for what I'm trying to do, then I would be inclined to request an improvement in how .deps.json is generated when PackageReference isn't involved, so as to have the same convenience for loading native libraries.

[wasabii]

I second this. Same issue. Building a .NET Core library that includes some native files (from a VCXPROJ). Works fine when I add it to the produced NuGet package: consumers generate .deps.json files referencing the libraries. However, doesn't work in a Console app within the same solution using ProjectReference, nor in a Unit Test. Though, it should. The only issue is that hte generate .deps.json files for those other projects do not fold in the native assets of the ProjectReference.

I would expect to be able to simply include Content (or None) item, and perhaps stick <AssetType>native</AssetType> onto it and have it pulled through.

[wasabii]

Digging into it, DependencyContextBuilder just seems to consider NuGet origins. Would need to fix that, as well as the associated GenerateDeps MSBuild task. And then the MSBuild targets to consider these items.

[chrisdill]

Adding to this to say I have the same issue with my project Raylib-cs. Took me a while to notice the .deps.json was not including runtimeTargets so for manual installs it would fail to find the library. I am also not sure how to workaround/fix this issue so any suggestions or more information is appreciated.

[vitek-karas]

@marcpopMSFT I think this is very much an SDK issue.

• SDK needs to communicate information about RID specific assets to the runtime.
• Such communication channel already exists, it's .deps.json (it's basically the main reason why this file exists today)
• SDK will correctly populate .deps.json if the RID specific assets are coming from NuGet packages
• SDK doesn't have mechanisms to define RID specific assets in non-NuGet sources (the project itself, dependent P2P references, ...) - and inherently it doesn't have a mechanism to correctly populate this information into .deps.json

[alexrp]

• SDK doesn't have mechanisms to define RID specific assets in non-NuGet sources (the project itself, dependent P2P references, ...) - and inherently it doesn't have a mechanism to correctly populate this information into .deps.json

FWIW, having a way to do this is essential for parts of vezel-dev/zig-sdk#8 as well.

[marcpopMSFT]

@vitek-karas Isn't this an issue in deps.json creation which from our discussion, the runtime team controls (hence our decision to move it to runtime repo)?

[jkoritzinsky]

@marcpopMSFT I agree with @vitek-karas that this is a dotnet/sdk issue.

The .deps.json format already has a representation for the required concept.

The task that generates the .deps.json is in the dotnet/sdk repo.

The .NET SDK needs to provide a (either add or document an existing) mechanism for developers to specify native assets for a given RID. This work is definitely SDK work as the experience is in the SDK itself, not in the runtime.

I'll let you two determine where this ends up. In any case, I think this falls into either Loader/Host or SDK since those are the areas that would own the discovery mechanism, not Interop.

[ghost]

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

TL;DR: How can I make DllImport automatically load the right native library from the runtimes/<rid>/native path when it is in a project referred by ProjectReference, similar to what PackageReference enables by adding the appropriate resolution information in deps.json?

--

If I create a NuGet package and put native DLLs in runtimes/<rid>/native/, the deps.json file generated for the assembly referencing that package will get runtimeTargets entries for the native libraries, allowing DllImport to find the appropriate DLLs to load. That mechanism is very useful for easily targeting multiple platforms with "Any CPU" despite reliance on native libraries that are each specific to a platform/architecture combination.

Now I would like to unit-test the classes in that package. Usually I put the unit test project in the same solution, make it refer to the project containing the classes to test. Unfortunately, that doesn't bring the mechanisms from PackageReference, as it only uses a ProjectReference. Therefore the deps.json for the unit test assembly doesn't get hints for the native libraries, and when the unit test is run, it complains about not finding the native DLLs.

A workaround would be to have a separate NuGet package containg nothing more than the native libraries, making the project containing .NET code refer to it, similar to what libgit2sharp does. However that's not convenient when updating the native libraries because it requires publishing their package before being able to test their integration in the .NET code.
I know about the NativeLibrary class where one can register a custom resolver, but that seems tedious for true multiplatform. I didn't find a simple way to replicate the RID-based resolution.

Is there any other way? Any thoughts?

Author:	antoinebj
Assignees:	-
Labels:	area-Host, untriaged
Milestone:	-

[vitek-karas]

While I agree that the final piece of code which writes this info into .deps.json can be owned by the runtime (although in the last several years we were not the owners of this), I think the primary problem is in defining the desired behavior. Which is in my mind solely an SDK thing. The SDK needs to define what is the right way to declare RID specific assets in cases others then PackageReference - this doesn't exist anywhere and it's not just about .deps.json writing, this information also needs to be correctly handled by things like RID targeted build/publish (when the SDK produces RID specific application, it should only copy the RID specific assets which are for that RID).

[marcpopMSFT]

@richlander has been digging into the RID behaviors and so may have an opinion here on how this should work long term.

[antoinebj]

Since this has been marked as a feature request, could you please advise on a better title?
"RID-based native library resolution for DllImport when using ProjectReference"?
If you also want me to put a clearer intro to the issue, feel free to advise.

[vitek-karas]

@antoinebj Thanks a lot for reminding us about this.

I think the core of the problem is lack of support for RID specific assets our project system. That is things like:

• How do I define a RID specific asset in a classlib/app (not NuGet package)
• ProjectReference should be able to consume classlib with RID specific assets correctly
• RID specific assets from all sources (app project itself, ProjectReferences and PackageReferences) are correctly copied to the output (for portable apps into runtime subdirectory, for RID specific apps into the root) and the correct .deps.json is produced.

I would suggest renaming this issue to something like "Support RID specific assets outside of NuGet packages" (or something similar).

I'll move this issue back to the SDK repo where it belongs.

[alexrp]

Is there a chance that this issue (or something like what @vitek-karas described) might be prioritized for .NET 7?

Having a way to include RID-specific assets combined with my upcoming work on project/package reference support in Zig.Sdk would be a game-changer for easily adding cross-compiled, in-box native components to any .NET app.

[saucecontrol]

I think the core of the problem is lack of support for RID specific assets our project system

I think the native binary resolution is just a symptom of a larger problem, which is that when consuming a project that builds a NuGet package as a ProjectReference, none of the package behaviors are brought in. That includes not just RID-specific binaries but also .props and .targets. Perhaps the answer is that things that are expressed by convention (file name/path, etc) in NuGet packages need to be able to be made explicit in the project file so they can be consumed by the SDK. That could also drive the package definition so that less of it is based on magic folder names.

[webczat]

just went straight into that issue. how is it to be handled now? build a local nuget just to reference it later?

[marcpopMSFT]

Moving to backlog. This relates to P2P references and native dependencies. Probably could use @richlander doing a pass through this as there isn't a good way to produce packages and projects that have native dependencies in a way that will work well for customers

[wasabii]

We're doing something to solve this in IKVM. But it involves our own P2P implementation, and build time modification of deps.json. so, not great.

[alexrp]

I'd really urge prioritizing this issue. .NET has an excellent interop story these days ... except when it comes to everything build-related. Resolving this issue would bring the story from excellent to amazing. It would also make life massively simpler for those of us who are maintaining MSBuild SDKs that bridge managed and native code.