Proposal: Release Nuget packages for .Net 4.7.1 version of System.IO and System.Runtime dlls

[johnthcall]

System.IO and System.Runtime both have NuGet version 4.3.0 packages which contain dll version 4.1.1.0 while .Net 4.7.1 has 4.1.2.0.
I have run into multiple issues with System.* NuGet packages where a Package A will depend on System.IO >= 4.3.0 but msbuild will unify the dependency to the GAC version. Is there a reason 4.1.2.0 dlls have not been released via NuGet?

[joshfree]

@ericstj @terrajobst

[4creators]

This applies to all inbox packages for .NET Core. The problem is that a lot of libraries reference older versions of System.**.* packages and .NET Core and .NET build systems do not treat this package references as referencing framework inbox assemblies and restores them with runtimes. Coreclr test build downloads unnecessarily almost 2,5 GB of packages and old runtimes while it needs only about 600 MB.

My proposal to partially fix it would be to publish all inbox packages as separate empty NuGet packages redirecting to appropriate framework - do not know all interactions with build system so it could be wrong idea.

Otherwise this could be fixed in SDK by explicitly forwarding all package references to target framework in case they are referencing older inbox assemblies published as NuGet packages.

[ericstj]

With respect to the OP comment. Those DLLs are no longer distributed via nuget. NuGet dependencies for framework components/netstandard had too many issues. We've gone back to the "inbox" model where 4.7.1 has reabsorbed all these DLLs back into the framework. You'll find the reference dlls provided by the targeting pack, just like any other inbox dll. The implementations will available in the GAC. It is by design that we'll ignore the package versions and unify to the targeting-pack version. Those DLLs are no longer distributed via NuGet. /cc @AlexGhiondea

@4creators: Even if we release new packages NuGet will still download the old ones as it walks the graph. So releasing newer versioned empty packages actually makes the problem worse since you'd now be downloading another full set of packages in addition to the older ones. This is why we did not ship new empty packages for things absorbed by netstandard2.0, it would only make the package graph problem worse. Instead we created build targets (conflict resolution) that could deal with the nuget (out of box) assets and inbox assets and choose the best one. Ultimately the package graph problem is solved by folks moving packages forward to no longer depend on the 1.x packages. That's why netstandard2.0 / netcoreapp2.0 was defined to not require package dependencies and assume api is provided by the framework.

Now I could imagine some NuGet feature that would short-circuit package download completely, but such a feature would have to come in NuGet as it is the one doing the graph walk and it provides zero control today of trimming the package graph (flags can impact asset resolution, but not dependencies). /cc @rrelyea @jainaashish

[johnthcall]

@ericstj: The only way I was able to resolve the build issue was uninstalling the 4.3.0 packages so there is no confusion by msbuild about which version of the dll to use. However this is a fragile fix as anyone adding another package with dependency on System.* will reinstall and require the team member to manually uninstall to get the build working. As you pointed out, if there was a way to avoid installing these packages in the first place that would be ideal.

[jainaashish]

@ericstj I don't completely understand the scenario and ask from NuGet. But just wondering why can't NetStandard library package have framework dependencies instead of package dependencies on System.*? Wouldn't that be able to solve the conflict resolution issue?

And allowing to impact asset resolution but not dependencies seems really fragile and vulnerable. It will end up creating more trouble then solving some edge cases or incorrect packages...

[ericstj]

@jainaashish the latest netstandard package effectively does that. The problem is with old packages. The goal would be to never download those packages on certain frameworks, so that the user doesn't need to download them just to get a placeholder file, or get the package assets removed by conflict resolution at build time. Imagine if .NETCoreApp (and .NETFramework) could say "I provide the equivalent API/runtime support for packages A:1.0.0, B:3.0.0, etc". This would tell nuget to never download those. This is effectively what is being done here, after nuget already downloads and resolves the packages. Currently there is no tell nuget to remove certain parts of the package graph before it does this work, even though we can tell it to ignore certain parts within a package.

@johnthcall it sounds like the SDK's conflict resolution wasn't completely working for you. If you have an isolated repro of System packages getting in way with .NET 4.7.1 please open up an issue with repro steps in https://github.com/dotnet/sdk. Please note that @dsplaisted recently made a few fixes that will make this better.

[KevinCathcart]

@jainaashish In case it helps, here is a slightly more concrete proposal of what I think Eric would like to see NuGet provide.

The basic idea would be that each framework could specify a set of packages with version ranges that they "provide", and for the purposes of dependency resolution any package version in that range is treated as already installed, and having no dependencies.

I'm not very familiar with the NuGet source, but I'd imagine this would be implementable as a special IDependencyProvider, similar to the project providers?

The tricky part part is where does NuGet get this information about provided packages from. The simplest solution is to have it baked in. Newer unknown versions of the framework would simply use the list from the latest known version, because frameworks don't unship APIs. (If they do, NuGet would already have an issue since it assumes that newer unknown framework versions are backwards compatible.)

For package based frameworks it might conceivably be possible to specify this information within the framework's main package, but that may mean downloading the framework package is required just to complete dependency analysis, which I think would be undesirable.

In any case the SDKs of the frameworks would be required to backstop this in whatever way is appropriate, so if the baked-in copy is out of date (and that framework really provides additional packages), it may download some packages that effectively don't get used, but things otherwise work.

This issue is not going to go away, and will likely get worse over time. Sure the .NET Framework may not be relying on NuGet packages anymore, but backport packages like the new System.Memory will still add to the set of packages that newer frameworks effectively provide. Being able to avoid downloading those when targeting a framework that has them built in is obviously very much desirable.

(In any case this should be discussed in an issue over on the NuGet repo.)

[jainaashish]

@KevinCathcart it doesn't seem that straight forward and like you mentioned there'd still be gaps to be filled. Like how frameworks would provide details about supported packages and versions to NuGet? How would these packages be tracked across multiple machines when installed frameworks are different..?, etc...

We can discuss it in person or a skype meeting to go further into the details and come up with the right solution.

cc @rrelyea

[KevinCathcart]

@ericstj My understanding is that the .NET Framework has 2 different sets of MSBuild files. One is the SDK version used by new SDK style project format. But there is also the classic MSBuild target files (from the msbuild repo).

I'd not be surprised at all to hear the users experiencing the most pain are those using the old style projects.. Looking into it, in my Visual Studio's MSBuild folder, the .NET Framework 4.7.1 does not enable AutoGenerateBindingRedirects. If you target 4.7.2 or newer, it does enable this property.

I'm not an expert in this, but with no automatic binding redirect generation, I'm not surprised that things fail here, because facades (unlike the core dlls) don't seem to get the special implicit binding redirection to the version shipped in the currently running framework. (I.e. how references to .NET Framework 2.0.0.0 assemblies don't error out in .NET Framework 4.X)

I've heard (but not verified) that the project template for exes are including AutoGenerateBindingRedirects set to true, but that does not help with retargeting existing projects.

I'd expect that this could be worked around by manually adding binding redirects, or setting this property to true in the project file, or by targeting 4.7.2.

This basic issue has been discussed in dozens of issues across the various projects. A lot of the discussion occurs on dotnet/standard#481, which first identified that we would have these sorts of issues. Unfortunately that thread is filled with lots of misinformation too.

The current situation is very unfortunate, but I don't think it can be fixed without potentially breaking some projects targeting 4.7.1 that are relying on that property being false by default. (e.g. if the project really does want to load two different versions of the same assembly strong named assembly from different locations).

@johnthcall The fix is probably to add the following to any 4.7.1 projects where you experience this issue. This has a pretty good chance of working if you are just referencing nuget packages based on net standard 1.X.

<PropertyGroup>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
</PropertyGroup>

Hope this helps.

[dsplaisted]

The following NuGet issue proposes a "supplied by platform" feature which could help address this issue: NuGet/Home#7344

[bartonjs]

This problem seems to be being addressed by tooling. Since there's no action for corefx at this time I'm closing the issue.