How to package .NET applications in Linux distributions?

[omajid]

A number of Linux distributions (and other OSs) are now using and packaging .NET (via source-build) into their distributions. With .NET available in the distro, users are looking to build on top of it. For example, someone was recently looking for guidance on how to package https://github.com/GitCredentialManager/git-credential-manager into Fedora. That led to some conversation in a Fedora context that I think is general enough that it would apply to most other distros as well.

You can see the original discussion here: https://pagure.io/packaging-committee/issue/1193.

I talked about this with @crummel who suggested starting a discussion here.

Before digging into specific, is this - packaging software built on top of .NET into distros - something we want to encourage and support?

If so, there's quite a few challenges that we need to solve. The challenges are similar to what we encounter in source-build: we need to build everything from source without access to the internet. Here's what I see as the big ones:

• Given an application that a user wants to introduce into the distro, we need to trace the entire dependency graph of the application and find some way to build it all from source, offline. What if there are cycles?
• Given a (dependency library's) nuget package, how do we find where the source code is and build it with the same configuration used by the nuget package author to create the nuget package?
• Given that foundational libraries like Newtonsoft.Json would be used by a huge percentage of .NET applications, it seems like it would fit in best with the distro policies if we packaged it up once and then applications or higher-level libraries could use that without having to maintain or build this code.
• If we are building these foundational libraries as nuget packages the distro, where should distros place these packages? How should applications consume these libraries at build time?
• What sort of build configuration/flags should we recommend? Should we tell users to use framework-dependent publishing or self-contained?
• Even framework-dependent deployments of .NET applications contain embedded third-party dlls. What do we do about this duplication on disk? There's also the issue that if a security issue is fixed in a library like Newtonsoft.Json, we would need to find a way to rebuild everything that consumes Newtonsoft.Json. Is there a way for an application to use a dynamic-linking or dlopen-style or LoadLibrary-style approach to load the packaged copy of Newtonsoft.Json.dll?
• What sort of paths do we recommend for applications and packagers to install .nupkg, .dll files that are meant to be used by other applications? Where should standalone applications live? (This can only be a very rough recommendation, because every distro has its own policies on paths and install locations).

This is also related to dotnet/core#7443

cc @dotnet/distro-maintainers @leecow @richlander @tmds

[jkotas]

Given a (dependency library's) nuget package, how do we find where the source code is and build it with the same configuration used by the nuget package author to create the nuget package?

The closest thing we have for this today is https://github.com/dotnet/designs/blob/main/accepted/2020/reproducible-builds.md . The basic idea is that the PDB of the binary in nuget package contains list of sources and compiler command line used to build the package that should be enough to allow re-creating the binary.

where should distros place these packages?
How should applications consume these libraries at build time?
What sort of build configuration/flags should we recommend?

How is this done for Java or Python? We should be able to follow the same plan.

[omajid]

Java is probably closest to .NET. The Java guidelines for packaging things in Fedora, for example, live here: https://docs.fedoraproject.org/en-US/packaging-guidelines/Java/

To answer the specific parts:

• Java libraries are placed at /usr/share/java/library-name.jar or /usr/lib/java/library-name.jar depending of whether it's architecture-neutral or architecture specific (path varies across distros, of course)
• Fedora includes a customized version of the java build tool maven called xmvn. This customized version works mostly like the standard tool except it knows to do a couple of distro specific things, like use /usr/share/java as java package repository to find libraries instead of Maven Central.
• Java doesn't really have Debug vs Configuration option. Self-contained also didn't really exist when the Java packaging systems for distros were being designed.

Python guidelines are here: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/

[richlander]

I like this general idea. It's a question of how to achieve it. While the reproducible builds are a good suggestion, it's definitely not productized and unlikely to be successful w/o cooperation from the library owner.

I propose some complementary ideas:

• Define prescriptive capabilities/requirements for source-buildable apps (and their dependencies).
• Work with common NuGet authors to play nicely with these capabilities/requirements.
• Gate inclusion on being source-buildable per above. This may require cutting off some functionality (IFDEF bad dependencies) to get included in a distro.

I'll be the first to admit that this is hand-wavy. Also, for such a system to work, there needs to be circular influence/feedback. This has the potential to be a big lift. Folks would need to really want this for it to work, including distros, app authors, and library authors. Certainly, it could work.

[mmitche]

@richlander We might be saying the same thing. Basically, if we took a look at the packages on NuGet, which ones are likely to be interesting to our partners, and what are their qualities? For instance, Newtonsoft.Json is interesting because it's an extremely common dependency in the .NET ecosystem. It contains multiple TFMs.

[ericstj]

Removing frameworks and RIDs can change the behavior of the package so it's not always "safe" to do. Back in the 1.0 days we had a switch that let source-build have "partial" packages where any removed configurations were replaced with placeholders - so that it would break the build rather than give someone the "wrong" asset. For example - imagine removing the netstandard2.0 asset but leaving a net45 asset - nuget would fallback to use the net45 asset. Another case would be removing netcoreapp2.0 but leaving netstandard2.0 and net6.0 -> a project targeting netcoreapp2.0 - net5.0 would start to get the netstandard2.0 asset instead of a netcoreapp2.0 asset which it might need. We have projects targeting all different frameworks throughout the stack (even in source build) so we'll need to decide how to handle this.

Technically it's not hard to build for different TFMs in source-build, it's just additional work. We can represent the TFMs with reference packs and the tools used to build them are purely managed. RIDS on the other hand are a different story since the managed components can (and do today) build on any platform, but the native assets require platform specific toolchains. So "worst case" I think we just overbuild the managed side.

I think the "best case" here is we standardize the entire stack on LatestTFM + netstandard2.0 - then we can remove all the other TFMs and not worry about behavior changes as a result. For RIDs we can also drop the assets specific to the RIDs we aren't building - similarly keeping in mind the heirarchy -- for example we might want to keep both linux and rhel when building for rhel.

It'd be nice if we could do this without forcing everyone to adopt changes to their projects. @ViktorHofer might have some ideas since this is similar to tricks we play in the runtime's TargetFramework SDK.

[mmitche]

@ericstj That's interesting insight. Any idea what happened with the placeholder approach?

[ericstj]

I believe it worked well enough, but over time we found so much of the product relied on building for so many different frameworks that we eventually just turned most of the build back on.

[tmds]

RIDs will require special care from the package maintainer as the native assets will either need to be source-build, or the managed library will need to be built so it uses the system provided libraries (for example: make libgit2sharp use system libgit2 instead of libgit2sharp.nativebinaries).

[tmds]

This is also related to dotnet/core#7443

@omajid what is the difference is with that issue? Can we keep a single issue?

[omajid]

I think dotnet/core#7443 is a subset. That issue that talks only about building applications from source/offline. That's a large chunk of this issue, but I wanted to capture the rest of the things that need to be done to onboard .NET applications into linux distros as well.

I am happy to merge this (though I don't know how to) if you think that we should only have one place tracking everything.

[tmds]

For me, the MVP here looks like:

Given a .NET project/solution, fetch all sources of its dependencies, and generate a build script to build the dependencies into a folder of nupkg that I can use to build the project/solution.

The sources and build script can then be used by a package maintainer to package up the app further.

[richlander]

That's a good MVP. As @jkotas suggested, we could use the PDB for that purpose. That seems kinda hard and also very indirect to me. It requires restoring the dependencies of the whole app first and then inspecting the PDBs in the nupkgs. Instead, it would be ideal if there was a top-level list of packages w/a link to the source repo.

It occurs to me that Central Package Management is super close to what we want. If we can add a Source property for each package and ask msbuild for that, then we have what we need. Yes?

https://devblogs.microsoft.com/nuget/introducing-central-package-management/

[tmds]

Each nupkg should have a reference to its source repo, ref, and maybe other metadata that describes how the package got built (e.g. .NET project path in the repo).

These sources will need to be downloaded for all dependencies, and the distro package maintainer will include them in the 'src package' that corresponds to the 'app package' he/she is building.

It's important to have the local sources resemble the upstream source repositories to facilitate patching.

The Central Package Management could be used to switch from using 'binary' packages from nuget.org to packages built from source locally.

[ayakael]

Has best practice evolved on this? I packaged powershell for Alpine Linux a year ago, and the newer versions now include more dependencies that are not trivial to build from source. An issue was opened with the upstream team: PowerShell/PowerShell#20143 but indeed I'm not sure how to proceed.

[Conan-Kudo]

We're also running into this with trying to bring Jellyfin into Fedora (rhbz#2238840). It'd be great to see some progress on this front.

cc: @mooninite