Change package testing to use RuntimeTargets rather than RID-specific restore

[ericstj]

Fixes #2017

@Anipik and I were playing around with this today and realized we were very close to enabling package testing using just single restore. I went ahead and finished our prototype. This should greatly reduce the restore cost and total duration of package testing.

@ViktorHofer this should unblock #53439

[ghost]

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes #2017

@Anipik and I were playing around with this today and realized we were very close to enabling package testing using just single restore. I went ahead and finished our prototype. This should greatly reduce the restore cost and total duration of package testing.

@ViktorHofer this should unblock #53439

Author:	ericstj
Assignees:	-
Labels:	area-Infrastructure-libraries
Milestone:	-

[ericstj]

To be more correct / replicate the actual runtime behavior here we would need to replicate the host selection logic to test with the right runtime assets for dependencies. It's not so interesting to do this type of testing since the SDK doesn't make it possible for a library to reference a runtime specific asset of another library or project, and we don't typically do this with our custom config system. It's bad practice for library to depend on implementation-only publics of another library; if one library needed it why wouldn't a customer library need it?

So the asset analysis ends up looking like this:

1. Package under test: Foreach runtime asset group
2. Package dependencies w/ exclude=comple: RID-less runtime asset group
3. Package dependencies w/o exclude=compile: compile asset group
4. Framework: ref-pack / reference assemblies only.

Note that 2-4 are changes from what we previously did, but these are "safe" changes, since if these validations succeed then API compat will ensure that transitively things should succeed.

[ericstj]

I broke this in a hilarious way. The structure of the package testing projects used to be

{packageId}\{TFM}\project.csproj

I removed the TFM since I can make the projects cross-target now that they don't need RIDs

{packageId}\project.csproj

When hexlix runs these, it's removing the packageId folder and placing the project in the root.

In the same directory as the packages folder. So it looks like this:

{workItemDir}\project.csproj
{workItemDir}\packages\...

Then what happens is all the package content gets globbed as None, and passed in as CandidateAssemblyFiles to RAR. CandidateAssemblyFiles happen to be considered first in RAR's precedence when resolving assemblies. So we were getting seemingly random assemblies resolved, based on whatever appeared first in the package cache (often not even appropriate for the framework).

Guess I need to restructure the helix test directories for these 😆.

[ViktorHofer]

As an alternative, you could also disable the globbing.

[ViktorHofer]

As you are removing some of the frameworkSettings files, why not remove the portable and netcore50 as well?

[ericstj]

As you are removing some of the frameworkSettings files, why not remove the portable and netcore50 as well?

I'm removing the ones that were no longer needed due to the change I made, since they were related to using the RuntimeIdentifier, or testing against runtime assemblies. I'll consider cleaning others if they're clearly dead code, but I don't want to feature creep this PR.

As an alternative, you could also disable the globbing.

I don't think that's necessary. Better to keep projects simple rather than mess with them to accommodate a wacky thing that customers should never do (putting packages restore folder under project).

[ericstj]

As an alternative, you could also disable the globbing.

I don't think that's necessary. Better to keep projects simple rather than mess with them to accommodate a wacky thing that customers should never do (putting packages restore folder under project).

So it turns out we can even delete the custom packages folder as Helix already does this for us:
https://github.com/dotnet/arcade/blob/1b17d7c9f0c2100bac5766b154c99fb31ab20eb4/src/Microsoft.DotNet.Helix/Sdk/tools/dotnet-cli/DotNetCli.targets#L21-L22

I'm looking into keeping the binlog, but only uploading it when there is a failure.

[ericstj]

PKCS test is failing due to dotnet/sdk#18129. I'm planning to change how I did the workaround for this. Previously I was relying on conflict resolution dropping a compile asset to indicate what package references should not be considered when evaluating closure of runtimes, but this doesn't work if the packages that should have been dropped were Exclude=compile. I'm either going to feed all the ref-pack assets into conflict resolution for consideration at runtime as well, or I'll leverage the PackageOverrides mechanism if I can get it to work.

[safern]

LGTM, thanks!