New Source Generator based approach to getting all the Win32 methods and types you need

[AArnott]

As announced in Making Win32 APIs More Accessible to More Languages, the Microsoft Windows SDK team now delivers a machine-readable metadata file that describes the entire Win32 SDK. This allows us to auto-generate C# p/invoke methods and supporting types rather than manually write them.

But even more interesting to users of this project, you no longer have to redistribute a bunch of PInvoke assemblies with your application. This deprecated model is stuck between not having enough of the Win32 SDK represented and having more than you app needs and bloating your shipping application. Instead, the new approach generates exactly the p/invoke methods and supporting types your app or library needs as source directly into your assembly. No bloat, and no extra runtime dependencies.

Check out the C# projection source generator that is based on the Windows SDK team's metadata over at microsoft/CsWin32.

So what happens to this project? Is there still interest in consuming pre-built assemblies via NuGet given this new alternative? We might continue to support and grow this project. We might augment or even replace its current hand-written code with the projection from CsWin32. What do you think?

Leave your comments below. Please look for thoughts you agree with and use the GitHub reaction buttons to vote up that comment rather than adding a new one, where possible. Feel free to vote down suggestions you dislike. We are entertaining all respectful suggestions here and are very interested in your feedback!

[AArnott]

Possible reason to maintain this repo: support .NET languages other than C# (since as of today, C# is the only language that supports source generation.

(vote this issue up only if it applies to you)

[AArnott]

Possible reason to maintain this repo: Avoid enabling unsafe code blocks in your own project.

(vote this issue up only if you are unwilling or unable to enable unsafe code in your C# projects)
See also microsoft/CsWin32#26 and consider voting that up if it would be meaningful to you.

[mythz]

Personally I'd still prefer consuming PInvoke nuget packages which I consider the official typed .NET API for Win32 APIs that works like a normal library that others packages can depend on to create & share higher-level functionality.

I do think the 2 initiatives should be unified so that it generates a source compatible API that projects can choose to easily move between source generators & pinvoke packages (if it's not already).

[ffMathy]

👍 I also think this project should use source generators to generate all the packages.

[AArnott]

It would certainly make maintaining this repo easier if we wholesale switched to the CsWin32 projection to build the nuget packages. There are two big caveats to this however, just to be transparent:

1. There would be an initial binary and source breaking change to the APIs offered in the nuget packages.
2. There would be an ongoing series of smaller breaking changes between releases because the metadata takes breaking changes to improve the API.

The second point deserves more explanation, so here it goes: there are still bugs in the metadata. There are missing APIs. And (more than anything) there are usability fixes to make such as moving sets of constants into enums. All of these changes potentially create breaking changes. When you are consuming a source generator directly in your project, life is good because you can directly manage when to update CsWin32 and thus when to deal with the breaking changes (if they impact you at all). But when you are consuming a .NET assembly, which generally speaking has to be shared across an entire app, breaking changes are much more difficult to deal with. Consider an app that consumes 3 libraries, each of which compiled against a different version of the PInvoke nuget packages. No matter which version the app ships, 2 of these libraries may fail at runtime because the runtime API doesn't match the compile-time API.

We could mitigate this somewhat by not jumping "all in" to 100% API coverage in this project using the projection. Instead, we could still add individual APIs at a time and check them for likelihood of future breaking changes and only add them to the package if they look stable. For example, we would reject additions from the projection that use constants that are likely to move into enums later.

[AArnott]

For you non-C# users, what would you say to the workaround of just defining a C# project in your solution for purposes of consuming CsWin32, and referencing that project from your F# (or other) language project?

[saul]

Another benefit to maintaining this repo as a fully precompiled CsWin32 is for users who do not have internet access in their dev environment but still want docs. From my understanding of the CsWin32 readme, you need internet access for it to be able to write XML docs:

Generates xml documentation based on and links back to docs.microsoft.com

[AArnott]

@saul no, the source generator contains a full copy of the docs in itself. No online access required.

[AraHaan]

I would love for it to be source generated as well (possibly also include MiniDumpWriteDump inside of dbghelp as well too as many people might want to call that api function as well). Currently Dbghelp package lacks the means of creating any sort of crash dumps manually in the code.

[AArnott]

This is the plan. It's happening in the CsWin32Projection branch.

[lostmsu]

@AArnott just an opinion, call it an educated guess if you will.

TL;DR; I am concerned about who will maintain Windows.Win32.winmd and CsWin32.

I wouldn't want this project to switch to CsWin32 until they stabilize it and maybe even receive the first patch, because I feel uncertain about CsWin32 future. This doubt is based on the fact, that .NET team which previously maintained WinRT/.NET (which CsWin32 is the replacement for) dropped it. While the real reason for that is unknown to public, I think it was because .NET team felt that supporting WinRT is too much of maintenance burden. If that is right, there's a risk that this project will have to maintain CsWin32.

Again, just guessing here, but I expect CsWin32 to be now maintained by the WinRT team. And I don't have much confidence in it, because WinRT so far looks like a major fiasco.

Now this all might be nulled if CsWin32 will not try to build an object-oriented COM-based API like WinRT did, and so will be much easier to maintain.

P.S. Arguably, this project itself already is a better Windows.Win32.winmd than the "official" thing.

[AArnott]

@lostmsu I'm glad you appreciate the work I and others have put into this project.

I am also on the "inside" 😉. I wrote CsWin32, and work closely with the team writing the metadata. While I have no crystal ball to say how long they'll be deeply investing in the area, they certainly are working hard on it now. And there seems to be a lot of good vibes going around that future updates to the Win32 SDK should be made in the metadata as part of each initial release so that it's maintained and even to influence how that API is designed with it in mind that it should express via the metadata in an API-friendly way. So in short: I'm optimistic.

My understanding of WinRT on .NET is simply that as a multi-OS runtime, it didn't make sense for .NET Core to maintain a Windows-specific feature like the WinRT dynamic bindings. CsWinRT is not only a response to that but one that it seems will help spurn a pattern for broader, high performance interop between .NET and other systems too.

CsWin32 is mostly about p/invokes and the structs/enums/delegate types that support them. There are (surprisingly to me at least) some COM interfaces that are in the Win32 SDK, and (maddeningly) some of these intermix with WinRT interfaces, which does indeed muddy the waters around where CsWin32 duties end and CsWinRT duties begin.

I believe and hope that the metadata will allow native APIs to be projected into .NET more exhaustively and accurately than we could do by hand here so that this repo can become a mere projection based on CsWin32. To that end, some of the niceties that are in this repo that would never come from the metadata are already being copied over as custom code fragments so we don't have to lose much if anything in the transition.

All that said, my own time spent on either of these projects has been scarce lately. So I wouldn't worry about things happening too quickly.

[Luiz-Monad]

What about a generic tool that parses C "extern" headers and make any P/Invoke work for the entire DotNet platform consume. Probably this will be as complex as the SWIG generator, also, its kind of hell to parse C/C++ headers, you basically have to dump the AST from the compiler in the middle-stages, because C++ parser is kind of Turing complete (yes, the parser itself, in C++, even the "eBNF" syntax is Turing complete).
But it would be so useful, it would solve a lot of problems with P/Invoke generation in the most generic way possible.
This kind of works, Python has a similar thing with its CTypes. Why don't we have one too here on DotNetLand ? (besides the effort)

[AraHaan]

What about a generic tool that parses C "extern" headers and make any P/Invoke work for the entire DotNet platform consume. Probably this will be as complex as the SWIG generator, also, its kind of hell to parse C/C++ headers, you basically have to dump the AST from the compiler in the middle-stages, because C++ parser is kind of Turing complete (yes, the parser itself, in C++, even the "eBNF" syntax is Turing complete).
But it would be so useful, it would solve a lot of problems with P/Invoke generation in the most generic way possible.
This kind of works, Python has a similar thing with its CTypes. Why don't we have one too here on DotNetLand ? (besides the effort)

are you refering to @tannergooding's ClangSharp that compiles C/C++ headers into C# p/invokes?

[Luiz-Monad]

What about a generic tool that parses C "extern" headers and make any P/Invoke work for the entire DotNet platform consume. Probably this will be as complex as the SWIG generator, also, its kind of hell to parse C/C++ headers, you basically have to dump the AST from the compiler in the middle-stages, because C++ parser is kind of Turing complete (yes, the parser itself, in C++, even the "eBNF" syntax is Turing complete).
But it would be so useful, it would solve a lot of problems with P/Invoke generation in the most generic way possible.
This kind of works, Python has a similar thing with its CTypes. Why don't we have one too here on DotNetLand ? (besides the effort)

are you refering to @tannergooding's ClangSharp that compiles C/C++ headers into C# p/invokes?

It already exists ! thank you.

[AArnott]

Also, we're already looking at ways to make our adapter for ClangSharp that creates the winmd, and CsWin32 which consumes that, available generally to solve such a problem.

[jarenduan]

According to my knowledge, IMHO, a combination of the 2 ways might be better, because:

1. PInvoke NuGet packages are intuitive, feeling like ordinary managed libraries. Consumers don't have to write an extra txt file to specify what APIs they'd like to use. They don't even have to remember the exact API name, as they could get IDE IntelliSence supports with packages. However, as you mentioned, deploying applications with these packages may have size or conflict issues.

2. CsWin32 based on code generation make fewer codes, which naturally bring a lot of benefits.

Thus, I think we should think up a certain mechanism to combine those advantages:

1. At consumer's development phrase, use packages for better experiences, such as IntelliSence, adjusted Enums, and etc.
2. When consumers build or publish the application, use CsWin32 to generate the just-enough pinvokes, and don't use packages.

I think it might be possible to achieve that goal. For example, we could automatically add win32 function name to the NativeMethods.txt when each time consumer calls the function in their codes.

Of course, all these are just primitive ideas. I hope it helps.

[AArnott]

Excellent points, @jarenduan. In fact we're trying to hit the mark in the middle as you describe in either of a couple ways:

1. With CsWin32: Offer IntelliSense completions for APIs even before you add them to NativeMethods.txt. When you accept the completion, we add the line to NativeMethods.txt for you.
2. Consume as a library, but with a toolset that automatically trims your app down to just those APIs your app requires when you publish.

Either of these approaches may help up get the best of both worlds. We don't have either of these yet though, so we'll have to see which arrives first and which provides the best experience overall.

[tannergooding]

Noting that consuming as a library/nuget can also be non-trivial. There are tons of circular references and layering obscurities that make exposing things into logical components difficult (this is true whether you try headers, DLLs, documenting technology groups, etc).

I have a lot of experience and have tried to work through that problem multiple times in TerraFX.Interop.Windows. I ultimately settled on exposing a single giant library (its about 15-20MB) and relying on trimming support for users to get it down to a reasonable size (typically 50-200kb, depending on how much you actually use).

CsWin32 helps a ton in the case where you only need a handful of APIs or types and is driven off Win32Metadata
Win32Metadata provides metadata for the entire SDK but needs tooling on top to actually be used in a language; it is driven off ClangSharp
ClangSharp itself allows fairly trivially generating blittable definitions for headers (supports netstandard2.0 and net6.0)

TerraFX.Interop.Windows (and Interop.Vulkan, Xlib, Libc, PulseAudio, etc) are examples of using ClangSharp to generate self-contained pre-built libraries that can be used as a "public surface area", rather than an "internal implementation detail". But that likewise can come with its own downsides/drawbacks.

Other libraries and projects are likewise using ClangSharp to help generate or drive their own bindings/scenarios.

[jarenduan]

With CsWin32: Offer IntelliSense completions for APIs even before you add them to NativeMethods.txt. When you accept the completion, we add the line to NativeMethods.txt for you.
Consume as a library, but with a toolset that automatically trims your app down to just those APIs your app requires when you publish.
Either of these approaches may help up get the best of both worlds. We don't have either of these yet though, so we'll have to see which arrives first and which provides the best experience overall.

@AArnott , That's cool! I'm looking forward to it. Great job!

[DaleStan]

CsWin32 doesn't appear to wrap the Win32 APIs into things that are sensible for .NET developers. For example, SetupDiGetClassDevs from this package returns an IDisposable, while CsWin32 generates a method that returns void*. In a similar vein, GetPrivateProfileString is generated as static uint GetPrivateProfileString(string lpAppName, string lpKeyName, string lpDefault, PWSTR lpReturnedString, uint nSize, string lpFileName), while this package would likely choose to include a static string GetPrivateProfileString(string lpAppName, string lpKeyName, string lpDefault, string lpFileName) helper method.

I appreciate having logically disposable things be actually IDisposable, and having helper methods that handle the ubiquitous call-allocate-call pattern.

[AArnott]

Thank you for your feedback, @DaleStan. The helper methods are still underdeveloped in CsWin32 and the helper overloads that you mentioned are great ideas that I think we could eventually generate in CsWin32 as well.

I also filed microsoft/win32metadata#930 to track the bad return type on the SetupDiGetClassDevs method.