Make DllExport attribute to be built-in into .net

[tapika]

Background and Motivation

On nuget packages there already exists 3 different packages which are dealing with [DllExport] attribute - that is 1. UnmanagedExports, 2. UnmanagedExports.Repack and 3. DllExport.

From all of them - 3rd alternative seems to be most up-to-date and coherent with time.

There are however some regressions coming from .NET platform itself, for example
3F/DllExport#158

Proposed API

Integrate [DllExport] attribute to be as a built-in standard in .net.

When talking about .net - I would prefer that functionality would come into .net framework, but if it's not possible - then .net core. (Up to you to decide).

Justification

• Programming language interoperability with .net family will only grow with time - we are talking about C++, Java - and also interoperability can be done between .net framework and .net core.
  (E.g. if main application is .net framework and takes long time to rebased under .net core - then DllExport could be potentially used as a solution).

• No need for buggy c++/cli debugging expirience - C++ and C# can communicate directly.

Known limitations

At the moment assembly where you inject DllExport - assembly needs to use specific cpu architecture - e.g. x86 or x64, but not AnyCPU. I think this is acceptable limitation.

[jkotas]

Take a look at the DNNE package published at https://github.com/AaronRobinsonMSFT/DNNE and provide your feedback there.

I would prefer that functionality would come into .net framework,

We are not adding new features to .NET Framework, except security and reliability updates.

Known limitations

The technique used by the DllExport package is Windows-specific that conflicts with our cross-platform portability goals for .NET Core. The technique used by the DNNE package does not have this limitation.

See #3750 for more details on this topic. Resolving as duplicate.

cc @AaronRobinsonMSFT

[tapika]

@jkotas @AaronRobinsonMSFT

I have briefly checked through DNNE, and idea is based upon having custom loader,

C:\Program Files\dotnet\host\fxr\3.1.4\hostfxr.dll

which is loaded by client before loading .net dll. (In your example DotNetLib.dll)

The idea is good, but that requires indeed using that custom loader, hostfxr.dll.

From programming language interoperability perspective it's much easier to just to load DotNetLib.dll, than to have intermediate loader for that purpose (hostfxr.dll).

With [DllExport] attribute it's possible to build .net dll, which is ready for use, rather than
trying to use custom loader for that purpose.

I suspect that indeed https://github.com/3F/DllExport might be limited to windows, but suspect solution could be ported to other OS's, like mac / linux.

Also it's much more easier to have [DllImport] <> [DllExport] attribute names reflecting each other, rather that [DllImport] <> [DNNE.Export(EntryPoint = "FancyName")].

So my proposal would be still to reopen this ticket.

Moreover - I want to have more functionality itself, and suspect it needs to be raised as a separate issue.

DotNetLib.dll:
[DllExport("TestMethod")]
void TestMethod()
{
}


HostApp.dll:
[DllImport("TestMethod.dll")]
public static extern void TestMethod();
...

// DotNetLib.dll is not loaded
TestMethod();
// DotNetLib.dll is loaded

// What function to call to free DotNetLib.dll, 
so .net runtime would know that we have released it ?

According to stackoverflow:
https://stackoverflow.com/questions/2445536/unload-a-dll-loaded-using-dllimport

It's indeed possible to force library to be freed, but .net runtime will be unaware of dll release.

See on stack overflow following comment:

You can then pinvoke FreeLibrary() twice to decrease the reference count to 0, passing it the IntPtr > you got from LoadLibrary(). That unloads the DLL, as well as any dependent DLLs that got loaded.

Beware that you'll get very nasty failure when you try to pinvoke any exported function on the DLL > again, any time after doing this.

[jkotas]

From programming language interoperability perspective it's much easier to just to load DotNetLib.dll, than to have intermediate loader for that purpose (hostfxr.dll).

We agree. DNNE generates a library that you just load. hostfxr.dll is internal implementation detail of DNNE that you do not need to worry about when using the native library produced by DNNE.

https://github.com/3F/DllExport might be limited to windows, but suspect solution could be ported to other OS's, like mac / linux.

The implementation approach used by https://github.com/3F/DllExport is very Windows specific. There is no way to port it to other OSes.

I want to have more functionality itself, and suspect it needs to be raised as a separate issue.

#12553

[tapika]

https://github.com/3F/DllExport might be limited to windows, but suspect solution could be
ported to other OS's, like mac / linux.

The implementation approach used by https://github.com/3F/DllExport is very Windows specific.
There is no way to port it to other OSes.

Currently how overall generation logic works - DllExport library uses ildasm and ilasm libraries to disassemble and reassemble dll. I think ildasm and ilasm should be also portable to other OS's - theoretically could be done out of box.

But if I would be in your shoes - I would consider modifying C# compiler itself, so it would be able to produce entry points suitable for other programming languages - so produced binary would be compatible with windows GetProcAddress and linux dlsym.

I'm also not sure about what kind of file format .net core dlls use. By quickly checking - I think they have 'MZ' header, refers that they are plain PE32 executables. Suspect on linux they work because dotnet is loading those .dll's, not native linux executables.

So theoretically
dlopen("DotNetLib.dll") is not possible to be done on linux itself.

But then - why wont keep PE32 executable format as such, only export function entries as normal windows executable (You will have then Dependencies tool supporting this idea) - and
to mac and linux only add functionality of loading PE32 executables.

(E.g. https://stackoverflow.com/questions/1548637/is-there-any-native-dll-export-functions-viewer )

It would indeed makes sense if .net core executable file format would be eventually documented and easily parsable/generatable in future by 3rd party libraries.

[tapika]

Btw, this could be considered as overlapping with:
#12405 (comment)

... dealing with multiple PDB formats, possibly writing a C++ parser for portable pdbs, and so on ...

So for mixed mode call stack we need to have PE32 parser, with potentially function name extraction. Why we cannot create v1.0 which would be portable with current plain C style function / windows / PE32 / logic, and then upgrade to support C++ class model bit better.

And for this task we need opposite operation - PE32 generator with function listing generation.

[AaronRobinsonMSFT]

I think ildasm and ilasm should be also portable to other OS's - theoretically could be done out of box.

These are already portable and built in the repo for cross-platform usage:

• https://github.com/dotnet/runtime/tree/master/src/coreclr/src/ilasm
• https://github.com/dotnet/runtime/tree/master/src/coreclr/src/ildasm

Suspect on linux they work because dotnet is loading those .dll's, not native linux executables.

That is correct.

It would indeed makes sense if .net core executable file format would be eventually documented and easily parsable/generatable in future by 3rd party libraries.

This is fully documented as the PE file format. The .cormeta section is stated to be undocumented, but the source for the parser is now available in .NET Core.

[tapika]

Btw, one thing still come to my mind.

If we code in C++, we still need to do this #ifdef-#ifndef:

#ifdef _WIN32
    h = LoadLibrary("test.dll");
    p = GetProcAddress(h, "TestFunction");
#else
    h = dlopen("test.dll")
    p = dlsym(h, "TestFunction");
#endif

Eventually after standardization (e.g. c++ standardization or os standardization) kicks in - then we will have at the end one function, which works on all OSs. Of course it might take long time, but we could take first step towards this.

On linux / mac we could have both - LoadLibrary and dlopen supported, with the difference that dlopen would work on ELF file format, but LoadLibrary would work on PE32 file format.

For linux LoadLibrary would be supported next to dlopen, but would work differently.

Then no need to have #ifdef complexity, and dll loading would always work directly for all os's:

    h = LoadLibrary("test.dll");
    p = GetProcAddress(h, "TestFunction");

Thinking bit deeper - suspect clang supports already PE32 file generation, for gcc it might take some time to implement. PE32 file loading logic is most probably done in wine, so if you want, you could even chat with wine developers to drag wine implementation out of wine make it a built-in into linux.

Btw, I have already tried to drag out dbghelp.dll from wine, might be non-trivial, and also protected by LGPL licensing, FYI:

http://wine.1045685.n8.nabble.com/dbghelp-detached-from-wine-into-standalone-windows-dll-td6007932.html

Eventually I've abandoned that approach, so there were not much comments or try-to-help from wine developers.

Stack trace is still handled via #ifdef hassle, see:
https://github.com/tapika/stacktrace/tree/develop/include/boost/stacktrace/detail

So dragging wine without wine developers support is no-no approach. But creating new implementation of LoadLibrary from scratch - I think it's completely doable.

Especially there exists libraries from which to take an example:
https://github.com/taviso/loadlibrary (Warning GPL licensing)

[tapika]

Btw - win developers already took some step towards file format harmonization, see my conversation with Alexandre Julliard,

Jun 06, 2019; 9:59amRe: dbghelp detached from wine into standalone windows dll...

It works if the .so file was built as a Wine library, with an embedded
PE header. It's not possible to load a plain .so lib, since there's no
way to return a valid HMODULE for it.

So wine developers modified gcc to be able to build linux executable with PE header as built-in into linux executable. Sounds like a hacky approach in overall, also not sure about implications of that decision - do we need to have wine emulator to run it ? So emulation is bit complex.

[tapika]

Fyi: https://stackoverflow.com/a/57534949/2338477

In all honesty, this solution seems to be quite complicated compared to the other DllExport answer. The C++ code is doing a lot of heavy lifting to scan for resouces and entry points, etc. One argument for this answer could be that it is cross-platform. However, the other DllExport answer is also cross-platform as well, and a lot simpler to use.