Support PInvoke exports via regular LoadLibrary/etc. without setting up runtime first

[MeikTranel]

As i understand, as of .NET Core 3.0 calling .NET library functions from something like a native wrapper is supported only via the new external host model, where a slew of APIs have to be called before actually being able to hit the .NET Core dll C API endpoint (but this time without the need for rewriting the assembly post-build).

I have been working on a modern replacement for the UnmanagedExports NuGet package, which reweaves attributed methods for VTFixUp entries and thus i am naturally interested. See
https://github.com/MeikTranel/NXPorts for more info.

Can we clarify whether it is a goal to support this feature in parallel to how it worked in .NET Framework.

[vitek-karas]

/cc @jkoritzinsky

[jkoritzinsky]

We currently do not have plans to support loading a .NET Core assembly via LoadLibrary the same way as in .NET Framework. It is possible to do by manually editing the PE imports table in the .dll file to point to the new ijwhost.dll instead of mscoree.dll (@batzen has been working on getting it to work for his SnoopWPF project), but it is not currently an officially supported scenario. Additionally, the temporary solution that @batzen has been working on and also the vtfixup table as-is will only work on Windows.

[Sewer56]

Related: dotnet/csharplang#308

Personally I would love this feature too, even if just only for Windows however not everyone can always have the cake they want. If there are technical reasons as to why exports cannot work on a multiplatform level (or simply would take considerable effort to do so) then I do accept the possibility of the developers not wanting to discriminate against other OS.

There are a lot of uses for such a feature in my opinion, but also dangers. An interesting use coming to mind is being able to write plugins/extensions for native applications in a managed environment which may be beneficial for many programmers who find greatly enhanced productivity in the .NET ecosystem.

An interesting danger is isolation. Implementing such an export mechanism should (probably)
isolate (or offer possibility to isolate) the individually loaded assemblies in their own AssemblyLoadContext(s) in case of multiple .NET DLLs with exports being loaded with different versions of same dependencies.

---

Random arbitrary thing: On Windows, it is currently not possible to get managed code execution in a native application in a synchronous way. If you do try, for example as given above write a plugin with a native library providing an interface for a managed component, the managed component couldn't be initialized inside of the native component's DllMain.

This is due to the Windows loader lock. Startup is still possible if you create a separate thread and allow DllMain to exit releasing the lock, then on subsequent exports check and wait (if necessary) for the managed component to finish initialising but ideally no programmer writing libraries/extensions should have the need to do that (most would probably not even understand the concepts/ideas in play).

Well... hacking around this is still possible, but very dangerous and non-recommended for anyone. Two ideas come to mind: Freezing all other threads and unfreezing after initializing managed code and hooking/detouring LoadLibrary variants and manual mapping to bypass the loader lock. Not ideal.

---

To random bypassers looking for alternatives.

If executing managed code in a native process is the goal, the officially supported and endorsed approach to doing to at the current moment in time with Core 3.X/.NET 5 (as OP mentions with external host model) is using the nethost and hostfxr libraries.

An example of this can be found on MSDN: https://docs.microsoft.com/en-us/dotnet/core/tutorials/netcore-hosting with samples available in the following Github repository https://github.com/dotnet/samples/tree/master/core/hosting .

The samples are relatively straightforward in my opinion, however one thing of note that I would add is that the signature of the managed function to execute needs to match the provided function signature in the example public static int Function(IntPtr arg, int argLength), else the hosting libraries will not return you a function pointer.

It's a bit of effort but you can still thankfully glue some C/C++/Other Native code to expose your .NET Library/Program to the native world.

Good luck!

[batzen]

I got it working in snoopwpf without a custom host and special setup on the "hosting" side, aside from loading ijwhost.
But my use case is a bit different than the usual use case as i have to load assemblies in a foreign process. And my, self imposed, limit is to only use managed code to achieve this.

The setup to create a working core dll also isn't that much different from the steps needed in full framework. The main difference is that i have to rewrite the dllmain of the core dll to point to ijwhost instead of mscoree.

It still does not work for self contained core apps because hostfxr currently prevents this scenario.

[vitek-karas]

@Sewer56 - just a comment on your suggestion that such load should involve some amount of isolation. Currently all the ways to load an assembly into a process (through the native hosting APIs in hostfxr) will already do that - the assembly is loaded into a new ALC which isolates its dependencies from the rest of the managed code in the process. As of now there's no way to disable this though - we were thinking about it, but didn't add a way to do that yet. Mainly because we were not aware of a scenario where it would be needed.

[Sewer56]

I see. Thank you.

I've never seen mention of isolation anywhere in the documentation so I assumed the behaviour was similar to NetFX/Framework where assemblies with exports get loaded into the default AppDomain. Glad to hear that I've been wrong.

[vitek-karas]

Not sure where in the official docs this is mentioned (if anywhere actually - we still have a lot of work left to cleanup the official docs) - it is definitely in the design doc: https://github.com/dotnet/core-setup/blob/master/Documentation/design-docs/native-hosting.md#loading-managed-components and for IJW: https://github.com/dotnet/core-setup/blob/master/Documentation/design-docs/IJW-activation.md#loading-the-assembly-into-the-runtime

[AaronRobinsonMSFT]

@MeikTranel I have created a project that might help with some of this. It requires the latest .NET 5.0 though.

It uses the nethost API and creates a native binary with the desired exports. It doesn't embed the managed assembly nor the JSON files, but does give you an example of how the system could work. You could imagine that using the Single File approach that this project could eventually produce a single native binary to deploy.

The project is at https://github.com/AaronRobinsonMSFT/DNNE and is also published https://www.nuget.org/packages/DNNE.

[MeikTranel]

I like C99 touch although I don't like installed dependencies, but that's something we can work around. What drives the sweat out of me is the idea of having to supply the marshalling, but I guess that was kind of bound to happen.
Maybe I can work on flowing my integration test suite into your in project, to help get the ball rolling - because tbh writing tests was about 99% of the effort I did on nxports so far.

One last question: have you considered the deployment scenarios yet? Single file standalone, single file framework dependent, regular multi dll output etc? Are there any theoretical barriers off the top of your head?

[AaronRobinsonMSFT]

I like C99 touch although I don't like installed dependencies, but that's something we can work around.

I assume you mean the requirements for VS or clang on the path?

If so, you can use the dnne-gen tool manually to generate the desired code and I could update it to copy over all the required source, libs, etc into the output dir so you can compile however desired. Feel free to file any suggestions/issues. These kind of scenarios are simple to support.

What drives the sweat out of me is the idea of having to supply the marshalling, but I guess that was kind of bound to happen.

As in you want the runtime supply that? If so, going the DNNE.ExportAttribute approach instead of the UnmanagedCallersOnlyAttribute should handle most of that. But then you need to supply a Delegate type.

One last question: have you considered the deployment scenarios yet? Single file standalone, single file framework dependent, regular multi dll output etc? Are there any theoretical barriers off the top of your head?

That is apart of the larger hosting issues. The self contained runtime et al is a hard problem and I am unsure where we are there for components - @vitek-karas could comment more. What would be nice is having the single file solution for libraries, not just applications. If that was done, then we could embed everything for the FDD scenario and have a single native binary.

[vitek-karas]

Single file framework dependent is definitely doable. .NET 5 should have support for creating applications like that, currently we don't have any plans to support components. That said I don't see any special problems with it. Especially when combined with the native exports per @AaronRobinsonMSFT 's project this could work very well.

Single file self-contained is very problematic. The plans are for .NET 5 to really only have a true single file for applications on Linux/Mac. Windows will for now still need the runtime to live in a separate .dll (limitations of the diagnostics stack if I remember correctly) but eventually we think it's also doable - for applications. Components are a different story though. The main issue is that we don't support loading multiple .NET Core runtimes into one process. Generic solution for self-contained components would have to allow that somehow. There are also problems with diagnostics (similar to the above) and couple of other cases.

Multi file should work just fine in all cases except self-contained components - same problems as mentioned above.

There's a discussion around a special case of a self-contained component: Basically the host only ever wants to load one managed component - in that case the issues with self-contained and multiple runtimes goes away. There's an interesting discussion on this topic here: #35465.

[dotnet-policy-service]

Due to lack of recent activity, this issue has been marked as a candidate for backlog cleanup. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will undo this process.

This process is part of our issue cleanup automation.