Create new packages to allow for side-by-side version installs and adjust naming for 5.0

[TraGicCode]

We should create the following new chocolatey packages to allow for people to install multiple versions of dotnetcore side-by-side.

dotnetcore1.0
dotnetcore1.1
dotnetcore2.0
dotnetcore2.1
dotnetcore2.2
dotnetcore3.0

If you don't do this then chocolatey can really only manage 1 version of the dotnetcore package at a time which is not ideal.

[jberezanski]

Fully agree. Given the servicing model of .NET Core, there should be separate package ids for each major.minor family. This would e.g. enable easy, automatic patching of the runtime in environments where you want to be most up to date and secure, but need to support a set of .NET Core families (build servers, hosting platforms such as Service Fabric) and/or remain on a given family even if it is not the newest one.

[jberezanski]

Now would be a good time to start working on this, given that previews of .NET 5.0 have started appearing.

[jberezanski]

Going forward, I would even consider dropping "core" from the package ids for 5.0+.

[jberezanski]

This approach could also make the package implementation simpler and more reliable, in cases such as when a given family introduces a new component or has different dependencies - this is very awkward to handle in the current AU streams-based approach.

[IanKemp]

Additional suggested packages:

dotnetcore2-lts - alias for dotnetcore2.1
dotnetcore3-lts - alias for dotnetcore3.1
dotnetcore-lts - alias for whatever the latest LTS release is (currently 3.1, so the package would currently point to dotnetcore3-lts)
dotnetcore - the very latest version (basically, the current package)

Further, how should the package identifiers be formatted?

dotnetcore3.1
vs
dotnetcore-3.1
vs
dotnetcore31
vs
dotnetcore-31

[IanKemp]

I'd be happy to take a bash at this, pending confirmation from the maintainers as to what packages they want and the naming formats. However I don't know anything about choco, so not sure how long it would take me.

[riezebosch]

Naming is the hardest part

[TraGicCode]

Naming is the hardest part

@riezebosch and @IanKemp

I would simply do it as i did in the original posting above.

dotnetcore1.0
dotnetcore1.1
dotnetcore2.0
dotnetcore2.1
dotnetcore2.2
dotnetcore3.0

each of the above represents a separate package. All security patches/bug fixes go into the correct package. For Example:

a security fix is done for 1.1. Therefore this version would have the following package versions

dotnetcore1.1
dotnetcore1.1.1 <--- or whatever the patch version is by the dotnet core team...

[riezebosch]

but then we have also dotnetcore-runtime, dotnetcore-runtime.install, dotnetcore-runtime.portable, dotnetcore-sdk, dotnetcore-runtimepackagestore,dotnetcore-windowshosting, ...

[jberezanski]

Here is my proposal for our course of action:

1. Drop support for .portable (zip) packages

As seen on https://chocolatey.org/packages/dotnetcore-runtime.portable#versionhistory, the download counts on various versions of dotnetcore-runtime.portable are very low, most below 200. The respective .install download counts are mostly in the 4000-10000 range (https://chocolatey.org/packages/dotnetcore-runtime.install#versionhistory).

Looking at this from another angle - the original incentive to publish both .portable and .install was based on the assumption that there would be a major difference in behavior: the native installer used in .install would overwrite the systemwide .NET Core instance with the current version (similarly to how .NET Framework 4.x behaved), while the .portable packages would provide the side-by-side ability. That assumption turned out to be false - there is no such thing as one systemwide .NET Core instance; all runtime versions can install side-by-side*.

(*As a user convenience, the installer for 3.x by default removes earlier patch versions of the same major.minor runtime family, which is normally a good thing, but can be turned off for some special scenarios.)

Therefore, in order to simplify the package names I would drop the .install suffix and stop publishing .portable.

2. Do not worry about out of support .NET Core versions

The new packages should be built for .NET Core 2.1, 3.1 and .NET 5.0. Other versions are out of support and we do not need to put any effort into making new packages for them.

3. Package families and naming rules

As discussed here earlier, each major.minor .NET version should be a separate package family. This aligns with default native runtime installer behavior (since 3.0) which removes older patch versions. From Chocolatey user standpoint, it makes choco upgrade useful again, because the command will stay on the same major.minor version. For maintainers, this gives the flexibility to easily take into account different dependencies and new installer switches between major.minor versions.

I would use these patterns for package names:
dotnetcore-X.Y-$subpackage (for 2.1 and 3.1)
dotnet-X.Y-$subpackage (for 5.0 onward)

With $subpackage being: runtime, sdk, windowsdesktop (3.1+), aspnetruntime, windowshosting.

All of these "subpackages" map naturally to Microsoft-published installers. As we know from experience, there is, unfortunately, some component overlap between the native installers (-sdk and -windowsdesktop include the -runtime; -windowshosting includes both -runtime and -aspnetruntime).

I prefer to include the dash before the version number for slightly better readability.

4. The Hosting Bundle and ASP.NET Core Module versions

There is one wrinkle regarding windowshosting (the "Hosting Bundle"). The unique component installed by it, not included in any other installer, is the ASP.NET Core Module (ANCM) for IIS. Contrary to the runtime and sdk, ANCM is in fact installed systemwide with no side-by-side capability. To complicate matters even further, .NET Core 1.0-2.1 used ANCM v1, while from 2.2 onward ANCM v2 is used. Both ANCM variants are independent of each other (install in separate places). Within each ANCM variant, each new version overwrites the previous one.

So, if we wanted to faithfully reflect the above in the packaging, we would need to craft two packages: one for ANCM v1, the other for v2, not aligned with the package families described above. This would, however, make maintenance harder (we would need to be on the lookout for possible new ANCM variants) and make choco upgrade unsafe (because it could install a newer runtime), so back to square one.

Given that ANCM versioning seems like an implementation detail, and that there is no separate installer for ANCM (only the Hosting Bundle, which is bound to a specific major.minor .NET family), let's just forget about these concerns and give -windowshosting the same treatment as other subpackages.

5. Metapackages

We have these currently:
dotnetcore (-> dotnetcore-runtime)
dotnetcore-runtime (-> dotnetcore-runtime.install)

The dotnetcore package can keep pointing to dotnetcore-runtime.
The dotnetcore-runtime package should point to the latest dotnetcore-X.Y-runtime, similar to how javaruntime points to jre8.
I would stop both packages at 3.1.x and create dotnet and dotnet-runtime for 5.0.0+.

There was also a suggestion for LTS metapackages; those could be named dotnet-lts-runtime and so on. However, I see them at lower prority.

[jberezanski]

Given the SDK versioning scheme described in https://github.com/dotnet/designs/blob/main/accepted/2018/sdk-version-scheme.md, the SDK packages should perhaps follow their own versioning, with each SDK feature release (A.B.Cxx) having its own package. This would, in fact, align with the current practice in the dotnetcore-sdk package, where we have separate streams for that.
Specifically:
dotnetcore-3.1-sdk-1xx
dotnetcore-3.1-sdk-4xx
dotnet-5.0-sdk-1xx
The dotnetcore-3.1-sdk package could then be a metapackage pointing to the latest feature package, currently dotnetcore-3.1-sdk-4xx.

I'm also toying with the idea of SDK metapackages reflecting support for various Visual Studio versions. For example, for 3.1, the relationship between SDK and VS version is:

SDK	VS
3.1.1xx	16.4
3.1.2xx	16.5
3.1.3xx	16.6
3.1.4xx	16.7

which is rather straightforward, but it might still benefit some users to be able to tell choco to install something like dotnetcore-3.1-sdk-vs16.4. Even more so with older SDKs which supported both VS 2017 and 2019.

[riezebosch]

I probably have time in the upcoming weeks te setup a few things. In my opinion the AU part is really valuable since it enabled us to keep up with the frequent updates with no maintenance. But I agree that specific packages for parallel installation would be really helpful so I aim to incorporate both.

[jberezanski]

Yes, I wouldn't dream of maintaining these packages without AU.
If life does not get in the way, I should be able to draft a prototype over the next couple of days.

[wertzui]

Just stumbling in from the discussion on the chocolatey page

Shouldn't .Net 5.0 be named net5.0 instead of dotnet5.0 to follow the target framework that is used in the *.csproj files?
That way the names could sync up with the names people are used to in their projects.
dotnetcorex.y
netx.y

[jberezanski]

The hosting bundle is a rather difficult customer.

It installs one unique component (the ASP.NET Core Module for IIS, ANCM), but it also installs the base runtime and the ASP.NET Shared Framework (what used to be called the Runtime Package Store, but its scope has grown, I guess). Those two latter components overlap with other packages (-runtime, -aspnetruntime), but they are optional - the user is able to opt out of their installation (https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/?view=aspnetcore-3.1#install-the-hosting-bundle).

There is no notion of "optional dependencies" in Chocolatey. A normal dependency (from -windowshosting to -runtime and -aspnetruntime) would be limiting: a user who does not need the runtime, for example, would need to tell Chocolatey to ignore dependencies, which is a rather hacky workaround. The existing hosting bundle package anticipated it and attempted to opt out of installing the runtime as part of the hosting bundle, reasoning that the user could always install the runtime via its own package - this unfortunately broke down when Microsoft changed the installation arguments and we did not update the script (#49). There was no provision in the existing package for opting out of installing the ASP.NET Shared Framework; we made the bundling explicit by adding a dependency on the -runtimepackagestore package (which generated its own set of problems related to moderation).

There are two ways, I think, we could approach the optional dependencies issue.

1. Granular packages:

• create a new package (-ancm) for the ANCM only (using all opt-out install arguments defined by Microsoft)
• convert the hosting bundle to a metapackage, which would depend on -runtime, -aspnetruntime and -ancm

The hosting bundle would then mirror the default behavior of the native installer. Users wanting a subset of the components would not use the hosting bundle package, but would have the ability to install each component separately, using the appropriate package.
This approach would be more aligned with the "package manager" way of doing things and users accustomed to working with Chocolatey would naturally know how to tailor the installation to their needs. However, it could increase the maintenance burden (we would need to closely follow the hosting bundle updates to ensure the opt-out switches still work; also Microsoft warns that there may be issues when the runtime is installed separately before the hosting bundle - we would need to investigate that and work around it).

2. Mimic the native installer behavior

• make the hosting bundle standalone, not dependent on any other package
• implement package parameters to support opting out of specific components

This would be less convenient to users (need to learn package parameters, not able to install this one together with other packages in one choco invocation, reduced dependency opportunities). On the other hand, it would reduce the risk and require less oversight and maintenance.

[jberezanski]

@wertzui That's an interesting suggestion.

I believe, however, that names such as netX.Y would be too generic as Chocolatey package ids - when viewed in the context of the entire Chocolatey ecosystem. The package naming guidelines strongly suggest to use the official name of the software. In this case, ".NET"/".NET Core" is somewhat special because of the dot, but virtually everyone I've heard pronounces it "dotnet", not "net". There is also the case of existing Chocolatey packages people are accustomed to (dotnetfx, dotnetX.Y, dotnetcore-*).

And the earlier Target Framework Monikers for .NET Core are actually netcoreappX.Y, which would look rather cryptic as a package id for people wanting to install the .NET Core runtime/sdk, not some unspecified "application". (Not all users are developers; there are also admins which do not need to know anything about TFMs.)

[jberezanski]

A couple of design matters to think about and decide:

1. Versions in dependencies
   Currently, the packages use strict, precise versions in their dependency expressions, such as <dependency id="dotnetcore-runtime.install" version="[2.1.23]" />. Because .NET uses only three version parts, we can freely use the fourth part for package fixes (e.g. 2.1.23.20201015). Due to the precise dependency versions, however, a fix for a leaf package causes a ripple effect - each dependent package suddenly needs to be updated (for no good reason, because the installed software remains the same).

   If we used dependency expressions with three parts fixed but the fourth open, we would still maintain the required consistency, but package fixes could be done to each individual package without impacting the rest of them.

   Examples: <dependency id="dotnetcore-runtime.install" version="[2.1.23.0,2.1.24.0)" /> or possibly <dependency id="dotnetcore-runtime.install" version="[2.1.23.0,2.1.23.99999999]" /> (less sematically correct, but slightly easier to implement).

2. The top-level metapackage (currently dotnetcore) was initially devised as an "all-inclusive" package, delivering all possible runtime components (including IIS integration). This was changed fairly early, because the IIS integration (dotnetcore-windowshosting, the Hosting Bundle) requires IIS for proper installation and many systems do not have IIS enabled, so having the most prominent .NET Core package fail on a large portion of systems was not making a good first impression. Currently the package simply points to the base runtime package.

   Since 3.0 we have, however, an additional runtime installer - the Desktop Runtime. I am wondering now whether the top-level metapackage should install:
   a) (option minimum, behavior unchanged) the base runtime, as it does now
   b) the desktop runtime (which includes the base runtime)
   c) (option maximum) all three runtimes: base, desktop and ASP.NET

3. The desktop runtime native installer includes the base runtime (same as the SDK and the Hosting Bundle). Should the package for the desktop runtime depend on the base runtime package, to reflect this composition - the fact that installing the desktop package also provides the base runtime? If not, then installing a third-party package which depends on base runtime on a system with desktop runtime installed may cause a needless installation of the base runtime - possibly with a newer patch version than the installed desktop runtime, which may lead to confusion.

   (I'm thinking: yes, it should.)

4. The -windowshosting dilemma outlined earlier - granular packages vs one package with parameters.

   Here I'm slowly leaning towards the granular packages option, if only to save users the hassle of having to learn and remember to provide the package parameters. The "missing runtimes" issue described by Microsoft should not be a problem here, because our runtime packages install both bitness runtimes in the proper order (first 64-bit, then 32-bit) so PATH should be set up correctly (to be tested, of course).
   The AspNetCoreModule does not install side-by-side, so there should be only one package id for it (with streams). The releases.json files provide the ANCM version in a version-aspnetcoremodule property.

[jberezanski]

Work in progress:
https://github.com/dotnetcore-chocolatey/dotnetcore-chocolateypackages/tree/dotnet5-and-redesign

MyGet feed: https://www.myget.org/F/jberezanski-chocolateypackages-dev/api/v2

[jberezanski]

Implemented:

dotnet-aspnetcoremodule-v2 - installs ANCM (v2) only (without runtimes), using the hosting bundle installer; package version is a combination of three parts of ANCM version (version-aspnetcoremodule in json), fourth part reserved for package fixes, and prerelease suffix taken from version-display of aspnetcore-runtime in json

dotnet(core)?-X.Y-windowshosting - metapackage with dependencies on dotnet(core)?-X.Y-runtime (fixed to specific major.minor.patch), dotnet(core)?-X.Y-aspnetruntime (fixed to specific major.minor.patch) and dotnet-aspnetcoremodule-v2 (right-unbounded)

[jberezanski]

Upon much thought, changed dotnet(core)?-X.Y-windowshosting dependency strategy on dotnet-aspnetcoremodule-v2 to be same as on runtimes - depend on specific major.minor.build version (with wiggle room on fourth segment to account for package fixes). ANCM does not follow semver (third segment is build number and does not reset with increase of minor or major), but this does not conflict with the dependency logic.

This makes -windowshosting installation safer and predictable, because it will not suddenly install ANCM from a later release.
The drawbacks (rather minor, I think) are:

• -windowshosting from multiple .NET major.minor channels cannot be installed simultaneously (conflict on ANCM dependency); users need to pick the ANCM channel they wish to use (typically the latest) and install individual runtime packages (-runtime, -aspnetruntime) from earlier channels;
• upgrading ANCM to a newer channel requires uninstalling the existing -windowshosting package (a noninvasive operation, since it is a metapackage).

Also - implemented dotnet-aspnetcoremodule-v1 (for 2.1) and dotnetcore-2.1-windowshosting. I will not bother with 2.0 because it has important info missing from releases.json, the hosting bundle installer has buggy parameter handling and 2.0 is out of support anyway.

[jberezanski]

Implemented dotnet(core)?-X.Y-sdk-Nxx and automation for generating new -Nxx SDK packages as part of AU run.

[jberezanski]

Restricting -windowshosting installation due to ANCM dependency conflict does not currently seem to behave as expected; possibly due to bug/shortcoming in choco (chocolatey/choco#116).

[jberezanski]

Built nupkgs for all 2.0+ runtime components (not including ANCM/windowshosting yet) and sdks released to date (except for those few with metadata missing in json) - available for testing on MyGet feed mentioned above. 306 nupkgs total.

[jberezanski]

Implemented metapackages: dotnet, dotnet-<component>, dotnet(core)?-X.Y-sdk. Removed 5.0 streams from dotnetcore-* packages.
Not much left to do:

• dotnetcore-desktopruntime (incorporating .NET Core Desktop 3.0.1 + .NET Core 3.0.1 Portable #46)
• [ ] converting existing dotnetcore and dotnetcore-<component> to metapackages using new tools and deprecating .install/.portable Due to the vast scope of changes being done, this will wait for phase two - let's publish the new packages first and iron out any kinks.
• testing... lots of testing

[jberezanski]

• dotnet-aspnetruntime
• dotnetcore-aspnetruntime

[jberezanski]

Here is a full list of new packages:

package id	notes
aspnetcore-2.0-runtimepackagestore
dotnet	(metapackage, has streams for Major.Minor 5.0+, points to dotnet-runtime)
dotnet-5.0-aspnetruntime
dotnet-5.0-desktopruntime
dotnet-5.0-runtime
dotnet-5.0-sdk	(metapackage, has streams for all 5.0 SDK feature releases, latest points currently to dotnet-5.0-sdk-1xx)
dotnet-5.0-sdk-1xx
dotnet-5.0-windowshosting	(metapackage, points to dotnet-5.0-aspnetruntime, dotnet-5.0-runtime and dotnet-aspnetcoremodule-v2)
dotnet-aspnetcoremodule-v1	(has streams for 2.1 and 2.2)
dotnet-aspnetcoremodule-v2	(has streams for all Major.Minor channels starting from 2.2)
dotnet-aspnetruntime	(metapackage, has streams for Major.Minor 5.0+, latest points currently to dotnet-5.0-aspnetruntime)
dotnet-desktopruntime	(metapackage, has streams for Major.Minor 5.0+, latest points currently to dotnet-5.0-desktopruntime)
dotnet-runtime	(metapackage, has streams for Major.Minor 5.0+, latest points currently to dotnet-5.0-runtime)
dotnet-sdk	(metapackage, has streams for Major.Minor 5.0+, latest points currently to dotnet-5.0-sdk)
dotnet-windowshosting	(metapackage, has streams for Major.Minor 5.0+, latest points currently to dotnet-5.0-windowshosting)
dotnetcore-2.0-runtime
dotnetcore-2.0-sdk	(metapackage, has streams for all 2.0 SDK feature releases, latest points currently to dotnetcore-2.0-sdk-4xx)
dotnetcore-2.0-sdk-1xx
dotnetcore-2.0-sdk-2xx
dotnetcore-2.0-sdk-3xx
dotnetcore-2.0-sdk-4xx
dotnetcore-2.1-aspnetruntime
dotnetcore-2.1-runtime
dotnetcore-2.1-sdk	(metapackage, has streams for all 2.1 SDK feature releases, latest points currently to dotnetcore-2.1-sdk-8xx)
dotnetcore-2.1-sdk-3xx
dotnetcore-2.1-sdk-4xx
dotnetcore-2.1-sdk-5xx
dotnetcore-2.1-sdk-6xx
dotnetcore-2.1-sdk-7xx
dotnetcore-2.1-sdk-8xx
dotnetcore-2.1-windowshosting	(metapackage, points to dotnetcore-2.1-aspnetruntime, dotnetcore-2.1-runtime and dotnet-aspnetcoremodule-v1)
dotnetcore-2.2-aspnetruntime
dotnetcore-2.2-runtime
dotnetcore-2.2-sdk	(metapackage, has streams for all 2.2 SDK feature releases, latest points currently to dotnetcore-2.2-sdk-4xx)
dotnetcore-2.2-sdk-1xx
dotnetcore-2.2-sdk-2xx
dotnetcore-2.2-sdk-3xx
dotnetcore-2.2-sdk-4xx
dotnetcore-2.2-windowshosting	(metapackage, points to dotnetcore-2.2-aspnetruntime, dotnetcore-2.2-runtime and dotnet-aspnetcoremodule-v2)
dotnetcore-3.0-aspnetruntime
dotnetcore-3.0-desktopruntime
dotnetcore-3.0-runtime
dotnetcore-3.0-sdk	(metapackage, has streams for all 3.0 SDK feature releases, latest points currently to dotnetcore-3.0-sdk-1xx)
dotnetcore-3.0-sdk-1xx
dotnetcore-3.0-windowshosting	(metapackage, points to dotnetcore-3.0-aspnetruntime, dotnetcore-3.0-runtime and dotnet-aspnetcoremodule-v2)
dotnetcore-3.1-aspnetruntime
dotnetcore-3.1-desktopruntime
dotnetcore-3.1-runtime
dotnetcore-3.1-sdk	(metapackage, has streams for all 3.1 SDK feature releases, latest points currently to dotnetcore-3.1-sdk-4xx)
dotnetcore-3.1-sdk-1xx
dotnetcore-3.1-sdk-2xx
dotnetcore-3.1-sdk-3xx
dotnetcore-3.1-sdk-4xx
dotnetcore-3.1-windowshosting	(metapackage, points to dotnetcore-3.1-aspnetruntime, dotnetcore-3.1-runtime and dotnet-aspnetcoremodule-v2)
dotnetcore-aspnetruntime	(metapackage, has streams for Major.Minor 2.1-3.1, latest points currently to dotnetcore-3.1-aspnetruntime)

[verzada]

Here is a full list of new packages:

Looks great, wasn't aware you guys already had spotted the need.

[jberezanski]

I think this can be finally considered done.