Include NetCoreAppCurrent configs in packages

[ViktorHofer]

Fixes #53854

The NetCoreAppCurrent configurations were omitted from packages to avoid
ever growing packages. Now that we adhere to the support policy for
packages we don't need to exclude them anymore as the policy defines a
baseline for .NETCoreApp configurations.

This will remove an artificial difference when source building the
repository and also make it so that partner repositories which don't
depend on a transport package like windowsdesktop (winforms) receive
the very latest assets that are included in the shared framework as
well.

[ghost]

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The NetCoreAppCurrent configurations were omitted from packages to avoid
ever growing packages. Now that we adhere to the support policy for
packages we don't need to exclude them anymore as the policy defines a
baseline for .NETCoreApp configurations.

This will remove an artificial difference when source building the
repository and also make it so that partner repositories which don't
depend on a transport package like windowsdesktop (winforms) receive
the very latest assets that are included in the shared framework as
well.

Author:	ViktorHofer
Assignees:	ViktorHofer
Labels:	area-Infrastructure-libraries
Milestone:	-

[ericstj]

I'm OK with this. Given the size of this and potential impact be on the lookout for changes downstream. I don't expect anything to break but you never know given this is shipping a lot of configurations which may never have been seen before.

[ViktorHofer]

@CertifiedRice was the approval intentional?

[ViktorHofer]

I had to update the packageindex as a few assemblies were missing a net6.0 entry which caused unnecessary package references (even for inbox libs).

[ViktorHofer]

@ericstj @joperezr can you please help me figure out how to react to the package system errors? It looks like the browser-wasm RID is trying to be restored on netcoreapp2.0 and above .NETCoreApp tfms but restore can't find the runtime packs for it. Simliar for ilumos, ios and others. Wondering if this is just the first time that we are packaging one of the new RIDs into a package?

[ericstj]

I believe this is an issue with how the test-tuples are calculated. cc @Anipik

Previously we had a set of known TFMs, and for each TFM we had a set of RIDs we knew it supported.

Looks like now the same set of RIDs are applied to all TFMs:
https://github.com/dotnet/arcade/blob/c7d6bd607715f334cda90e01967bb0c02dee09be/src/Microsoft.DotNet.PackageTesting/GetCompatiblePackageTargetFrameworks.cs#L37-L38
So it's applying newer RIDs to older TFMs, thus these errors when validating.

That task should be changed to only apply those RIDs to the TFM of the asset they apply, or compatible TFMs (to catch a mistake where RID-specific was used in early framework, but dropped later: package would still resolve RID specific on newer framework).

[ericstj]

It's really unusual for us to have this many RID-specific assets in a package. It was bad before, but this makes it worse.
Anything we can refactor to make this simpler?

[ViktorHofer]

Opened #53900 to discuss this in more detail.

[Anipik]

I think it will need to if we don't want to give folks different behavior. Once we start removing the "special" characteristic of NetCoreAppCurrent then folks will no longer try to catch differences between it and other TargetFrameworks: that's a key benefit we're getting here.

@ericstj @ViktorHofer can you elaborate on the advantages of doing this ?

Requiring accumulation of TargetFrameworks will result in some churn to the codebase, modifying conditions, ifdefs, etc. It will make turning the crank on versioning in the repo a bit harder (hopefully tool-able).

probably with the correct patterns, we should be able to reduce it. However, we should try to make the policy concrete and hopefully not change it as we spent a lot of time during last releases as well to automate the previous process.

[Anipik]

That task should be changed to only apply those RIDs to the TFM of the asset they apply, or compatible TFMs (to catch a mistake where RID-specific was used in early framework, but dropped later: package would still resolve RID specific on newer framework).

i can try to fix this later today.

[ericstj]

Looks like we are including browser here just for this little ifdef:

runtime/src/libraries/System.Text.Encodings.Web/src/System/Text/Encodings/Web/OptimizedInboxTextEncoder.Ascii.cs

Lines 25 to 35 in 85fde05

	#if NETCOREAPP
	#if !TARGET_BROWSER
	[FieldOffset(0)] // ensure same offset with AsBytes field
	internal Vector128<byte> AsVector;
	#else
	// This member shouldn't be accessed from browser-based code paths.
	// All call sites should be trimmed away, which will also trim this member
	// and the type hierarchy it links to.
	internal Vector128<byte> AsVector => throw new PlatformNotSupportedException();
	#endif
	#endif

Doesn't seem worth it to me. Also I don't like us having PNSE's buried in a library that has nothing to do with platform-specific functionality (Encoding). Shouldn't we have some runtime feature flag that can detect if the platform supports this path and avoid even going down here? @eerhardt @GrabYourPitchforks

[eerhardt]

Shouldn't we have some runtime feature flag that can detect

The issue is the [StructLayout(LayoutKind.Explicit)] attribute on this type. If the linker sees a used struct with that attribute, it can't trim any of the fields on it - since you are defining a StructLayout, you are probably using it in a P/Invoke, so trimming fields would break the P/Invoke.

Since the field can't be trimmed, it means that Vector128<T> can't be trimmed in browser-wasm.

[eerhardt]

FYI - @vitek-karas @MichalStrehovsky @marek-safar. Looks related to dotnet/linker#1309. There is some discussion there about this exact scenario.

[ericstj]

since you are defining a StructLayout, you are probably using it in a P/Invoke, so trimming fields would break the P/Invoke.

I believe @GrabYourPitchforks did this so that he could have a union structure where he could access the same bytes as a Vector as well. Is there a way to annotate that it's safe to trim it? Alternatively the linker could identify that this is a union, where it's safe to trim a field so long as the size of the struct remains the same.

[marek-safar]

I believe @GrabYourPitchforks did this so that he could have a union structure where he could access the same bytes as a Vector as well.

Can this use Unsafe.As or similar at the call site instead of explicit Vector128 inside the struct?

[MichalStrehovsky]

Can this use Unsafe.As or similar at the call site instead of explicit Vector128 inside the struct?

The presence of Vector128 changes the alignment requirement of the struct so dropping the field and using Unsafe.As would likely result in unaligned reads.

Looks like the problem was that the presence of Vector128 regresses the WASM size by ~4kB compressed. Vector128 is more fat than it needs to be. I assume it's caused by Equals/GetHashCode/ToString having too many dependencies.

[ViktorHofer]

I opened #53900 to discuss this in more detail.

[ViktorHofer]

@ericstj @ViktorHofer can you elaborate on the advantages of doing this ?

Shipping the very latest and greatest .NETCoreApp asset (which in some cases also ships via the shared framework) guarantees that when the package evolves, optimizations that only apply to the latest version of .NETCoreApp aren't lost.

probably with the correct patterns, we should be able to reduce it. However, we should try to make the policy concrete and hopefully not change it as we spent a lot of time during last releases as well to automate the previous process.

I agree with you on that. Using the right patterns reduces churn by ~90% but the remaining 10% will need manual action:

• TFM updates (could be automated by deferring to a property which specifies the NetCoreAppPrevious tfms, i.e. for net7: net6.0 and for net8 net6.0;net7.0).
• References or conditions that apply to a previous not built tfm anymore. We want to clean this up with every release to not add on top of each other.

FWIW, the convergence of ref and source project would definitely help with keeping tfms in sync and reducing the number of conditions that apply to specific tfms.

I agree on that we either need documentation or make versioning of packages trivial to mimic.

[ViktorHofer]

i can try to fix this later today.

Thanks a lot @Anipik. I will meanwhile label this PR as blocked.

[joperezr]

Is there a design doc or discussion on what the plan is going forward for assets in terms of support? My questions are mostly around the following scenario: library System.Foo has configs (netstandard2.0;netcoreapp3.1,netcoreappcurrent) in the package, and then netcoreapp3.1 goes out of support. Does that mean that we drop netcoreapp3.1 asset from the package? do we add a new configuration for the dropped config +1?

[ViktorHofer]

Is there a design doc or discussion on what the plan is going forward for assets in terms of support?

Yes, see @GrabYourPitchforks's design doc which isn't public yet: https://microsoft.sharepoint.com/:w:/r/teams/DotNetTeam/_layouts/15/Doc.aspx?sourcedoc=%7B0176D4ED-B6F9-4C9D-95E2-DDE2BCC00688%7D&file=Addressing%20dotnet%20OOB%20servicing%20pitfalls.docx&action=default&mobileredirect=true&cid=92b19bfc-c3b6-41aa-93ab-67d2208b5946. Note that the relevant section just enumerates the package policy that is already in place but which just wasn't enforced until recently.

To provide you an example of what's going to happen for NET7 and NET8 see the following:

• When main targets NET7, packages will drop the netcoreapp3.1 and net5.0 assets and contain the minimum supported .NETCoreApp configuration net6.0 (LTS) and net7.0 (Current).
• When main targets NET8, .NETCoreApp packages assets won't be dropped as net6.0 (LTS) will still be supported when NET8 ships but net8.0 (Current) will be added.
• When main targets NET9, net6.0 and net7.0 will be dropped, net8.0 (LTS) will be kept and net9.0 (Current) will be added.

By only including tfms which are supported by the "current band" (i.e. NET6) an infinite growth in package size is prevented. In the past NetCoreAppCurrent configurations were excluded from packages as there was no lower boundary that was enforced. Until just recently we even carried portable-* and netcore4/5 assets along even though those are out of support for quite some time.

[ViktorHofer]

Encountered conflict between 'Platform:System.Formats.Asn1.dll' and 'CopyLocal:C:\h\w\B0C609B1\w\AC1C095C\e\.nuget\system.formats.asn1\6.0.0-ci\lib\net6.0\System.Formats.Asn1.dll'. Choosing 'CopyLocal:C:\h\w\B0C609B1\w\AC1C095C\e\.nuget\system.formats.asn1\6.0.0-ci\lib\net6.0\System.Formats.Asn1.dll' because file version '42.42.42.42424' is greater than '6.0.21.20104'.

During ResolvePackageFileConflicts we are feeding in the shipping SDK (currently 6.0 Preview 3) with valid assembly file versions. In non official builds, the file versions are set to 42.42.42.42424, mostly for convenience. Because of that, conflict resolution prefers the copylocal asset from the nuget package instead of the inbox asset.

Opened #53854 as well to discuss this.

[ViktorHofer]

@joperezr just FYI, VerifyNotDependsOnNetStandard wasn't running for the last year as the internal properties that it depended on were removed in the SDK.

[safern]

To provide you an example of what's going to happen for NET7 and NET8 see the following:

I guess for packages that support NS2.0 or NS2.1, we are going to add targets that error when an out of support tfm is used to restore the package?

[ViktorHofer]

I guess for packages that support NS2.0 or NS2.1, we are going to add targets that error when an out of support tfm is used to restore the package?

Yes and that's already implemented (and automated). See 293d472 for more details.

[ViktorHofer]

Merging as CI seems to take very long today. The previous run was green and the allconfigurations leg (which is the only code path affected by the last commit) already passed.

[safern]

Yes and that's already implemented (and automated). See 293d472 for more details.

Awesome, thanks for sharing as I missed it because it happened while I was gone.