Question / Suggestion: Allow managed exports (UnmanagedCallersOnly) in non AOT scenarios.

[LostTime76]

I am a high level user of .NET and am not intimate with the implementation details. Is there a reason why it seems UnmanagedCallersOnly is only being applied to AOT scenarios? I need it for non AOT scenarios - managed DLL exports.

I have a use case where I am hosting a native runtime in my managed application by using P/Invoke to load the native runtime DLL and functions. However, the native runtime itself also has the ability to P/Invoke into native libraries. I want to expose some functions in my running managed exe / DLL as native functions so that the runtime I am hosting can use them. So.. reverse P/Invoke?

There are a few ways I could see this working or be implemented.

When we build C# .NET projects nowadays, we end up with a dll for the code and an exe stub that just runs some main function within that dll to bootstrap the program. We could allow native exports directly from a managed dll provided they follow the UnmanagedCallersOnly rules. The native runtime can then just get a handle to the dll and pull out the managed function using the export.

For my scenario, I started by trying to make a native AOT glue library that referenced my managed main exe project. Basically like this:

main exe / dll -> native runtime -> glue library -> main exe / dll

I had thought the native AOT glue library would have just referenced the main exe / dll somehow during runtime. The issue is that the glue library now self contains all of the code in the main exe / dll and thinks its going to run independently. When I debugged, everything worked fine and the native runtime accessed the glue library which executed the functions in the main exe / dll. However, I thought it would be executing the functions inside the running process, so I would hit a breakpoint on managed code within the main exe/ dll but I did not. This is due to the glue library containing its own copy of the managed code in the main exe / dll.

If we stick with the AOT scenario here, I am not sure how you would get it to work to reference the managed main exe / dll, but at the least, I don't want it containing a copy of the code within the main exe / dll like it does now, because its not running "independently".

My vote would just be to allow native exports from the managed dll. This way, we don't have to create a separate project and can just export directly from a single dll. Is there some technical reason this cannot be done today?

For reference, I can (and have) implemented some test code to basically expose static managed functions to the native runtime using reflection to get the static function method handles and pointers and it works fine. However, its a bit of a cludge whereas just allowing to export those functions directly would be provide clearer intent and less code. If its possible to do this at runtime anyways, why can't it be done with the UnmanagedCallersOnly export?

[jkotas]

Take a look at https://github.com/AaronRobinsonMSFT/DNNE. It creates the wrapper native library with the named UnmanagedCallersOnly exports.

[jkotas]

cc @AaronRobinsonMSFT

[AaronRobinsonMSFT]

@LostTime76 Please let me know if have any questions about DNNE. We use it in the this repo to simplify interop testing scenarios and people have been able to address scenarios similar to what you describe above.

[LostTime76]

@AaronRobinsonMSFT

I have tried the library. It is producing an extra dll for my solution called tstNE.dll. It looks like when I host the native runtime it is calling back into the "main exe / dll" of the hosting process like I want. It is kind of an annoyance that I would have to distribute an extra dll with this solution, but I think it gets me what I want for now.

Are there any plans for .NET to be able to do this directly like I suggested above (non AOT scenarios) instead of having to use the external nuget package and produce an extra dll like this? Perhaps an improvement as the .NET AOT / interop functionality is further developed? Didn't some of the AOT stuff start out as an external nuget package like this and eventually it was integrated directly into the .NET runtime package?

[LostTime76]

@AaronRobinsonMSFT

Wait.. I am perusing the github source for the DNNE library. It looks like the dll is creating its own CLR instance, which would not be what I want. Is the code getting the CLR instance of the host that called the DLL?

Maybe I am just better off doing the cludge of passing a block of function pointers retrieved at runtime if there are no plans to actually support this scenario properly.. even though I am not sure why it can't be done by just exporting directly from the managed dll.

[AaronRobinsonMSFT]

Are there any plans for .NET to be able to do this directly like I suggested above (non AOT scenarios) instead of having to use the external nuget package and produce an extra dll like this?

No plans and unlikely to ever be an option. Creating native exports directly on a managed assembly and making it cross-platform is very difficult - it is what C++/CLI was for. C++/CLI is supported on .NET Core, but only on Windows. If that is an acceptable limitation then this is possible.

Didn't some of the AOT stuff start out as an external nuget package like this and eventually it was integrated directly into the .NET runtime package?

Yep. We've discussed integrating DNNE into the platform several times. At present it doesn't seem to be a big enough community need to justify the work though. Especially when it is as simple as a NuGet reference.

It looks like the dll is creating its own CLR instance, which would not be what I want. Is the code getting the CLR instance of the host that called the DLL?

Yes. It is either activating a CLR instance if one does exist, or reusing an already loaded one.

[AaronRobinsonMSFT]

if there are no plans to actually support this scenario properly

I don't think this is a very fair characterization. Simply because there is an misunderstanding of how the system works and the inherent complexity of hosting/activation doesn't mean the scenario isn't "proper". By your own admission the following:

I am a high level user of .NET and am not intimate with the implementation details.

If this is the state of things, then I find any statement to qualify something as "proper" is unwarranted.

There are three approaches to performing the action in a reliable manner.

1. C++/CLI - but only on Windows

2. Manual hosting - similar to what DNNE does. There are multiple ways to handle this, but the APIs used will be the same.

3. A bespoke solution that makes assumptions about how and when the runtime is activated - similar to the approach you are currently taking.

None of them are better than the other, it all depends on what is trying to be accomplished.

[LostTime76]

I am not speaking ill will towards any of the work you or the .NET team has done. That's not my intention. I apologize if it sounded that way.

In my opinion, "proper" would mean just simply allowing the export directly from the managed dll without all of the extra complexity and steps. It seems like this is very possible based on my own findings with just passing the function pointer at runtime and stuff like DLLExports which seems to just modify the IL directly. Whether that's simple to do in the runtime, I don't know, but it looks like all the tools are there, there is no technical reason it can't be done, and is just a matter of putting in the effort.

Maybe I should have changed the wording from "proper" to "supporting this in a simple way that just exports directly from the managed dll".

[LostTime76]

@AaronRobinsonMSFT

I missed your previous commet:

Yes. It is either activating a CLR instance if one does exist, or reusing an already loaded one.

So this means it would be using the CLR instance that is already running from my process when my "native runtime" calls back into my "main exe / dll", and does not actually create a new CLR host instance, correct?.

You make a good point about the cross platform stuff. I technically only intend for my stuff to run on windows, but I don't want it limited to only windows if the case ever arises. I think the better option here for now would be just to do the static function runtime pointers passed to the native runtime.

Thanks

[AaronRobinsonMSFT]

So this means it would be using the CLR instance that is already running from my process when my "native runtime" calls back into my "main exe / dll", and does not actually create a new CLR host instance, correct?.

Correct.

You make a good point about the cross platform stuff.

Note that DllExport and many other native export tooling is limited to Windows. The approach taken by DNNE, using the extra native binary, is currently the only cross-platform approach. The ECMA-335 export mechanism that DllExport and C++/CLI uses is limited to Windows for a myriad of reasons that aren't worth getting into.

I think the better option here for now would be just to do the static function runtime pointers passed to the native runtime.

Okay. This is a fine approach and if you completely control your applications ecosystem then it is a great option.

I'm going to close this issue now. If you have any follow-ups you can post them here or feel free to create a new issue.

[AaronRobinsonMSFT]

@LostTime76 If you would like to learn more about the limitations of the DLLExport approach and why the pattern that DNNE follows is recommended, the following issue is informative - #37556.