[System.Runtime.Loader] Add hot reload test infrastructure

[lambdageek]

Adding infrastructure for hot reload testing.

For each test we define a new library assembly project. The .csproj has a
DeltaScript property that specifies a JSON file that lists the name of an
initial source file, and a list of updated versions of that file. The
hotreload-delta-gen tool runs during the build to read the delta script and
create deltas that incorporate the updates.

The main testsuite references all the test assemblies, and when a test runs, it
calls ApplyUpdateUtil.ApplyUpdate to load subsequent deltas
and then compares the results before and after an update.

Dependencies:

• https://github.com/dotnet/hotreload-utils - there is now a Microsoft.DotNet.HotReload.Utils.Generator.BuildTool MSBuild targets nupkg on the dotnet6-transport nuget feed.
• If DOTNET_MODIFIABLE_ASSEMBLIES is not set in the environment, we use remote executor to run the tests.

Mono is technically enabled, but practically it's not running anywhere:

• We have not enabled the MonoMetadataUpdate support on desktop linux or macos
• For wasm, ios and Android since we don't have remote executor support, we would need to adjust the test pipelines to pass the DOTNET_MODIFIABLE_ASSEMBLIES flag via xharness (and the app builder tasks).
• For ios and Android we would need to enable the interpreter as the execution engine, otherwise the tests are skipped.

To try it out locally with CoreCLR run:

./dotnet.sh build src/libraries/System.Runtime.Loader/tests /t:Test

For mono, build the runtime with /p:MonoMetadataUpdate=true and setenv MONO_ENV_OPTIONS=--interp

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[lambdageek]

@tommcdon @mikem8361 So here's the general shape of how I think hot reload tests should look.

@akoeplinger @ViktorHofer I really need help setting up arcade stuff for https://github.com/dotnet/hotreload-utils - the end state is that I'd like for hotreload-delta-gen to be a dotnet tool that's packaged in a transport package that dotnet/runtime could pick up using arcade/BAR/maestro. I tried setting up the eng/ folder, and Alexander set up an internal repo for the publishing. But there's no pipeline that pushes out packages yet.

@akoeplinger @ViktorHofer I also will need some help with CI. The tests depend on some environment variable being set (DOTNET_MODIFIABLE_ASSEMBLIES=debug) and I'm not sure if we should set that in the pipeline, or if there's some way to have the test project itself start up with environment values set.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Adding infrastructure for hot reload testing.

For each test we define a new library assembly project. The .csproj has a
DeltaScript property that specifies a JSON file that lists the name of an
initial source file, and a list of updated versions of that file. The
hotreload-delta-gen tool runs during the build to read the delta script and
create deltas that incorporate the updates.

The main testsuite references all the test assemblies, and when a test runs, it
calls ApplyUpdateUtil.ApplyUpdate to load subsequent deltas
and then compares the results before and after an update.

Dependencies:

• https://github.com/dotnet/hotreload-utils the hotreload-delta-gen binary
  must be installed and on the PATH (TODO: package it as a dotnet tool and
  publish to a transport nuget package)

Needs work:

• Mono test runs need to pass the FEATURE_MONO_METADATA_UPDATE property to
  msbuild to the test project. This is because not every mono configuration
  includes hot reload support. Eventually this should be subsumed by the
  runtime API to querty hot reload capabilities internal runtime API to return hot reload capabilities #50111
• All runs need to pass DOTNET_MODIFIABLE_ASSEMBLIES=debug to the testsuite.
  Hot reload only works when:
  • the assemblies are compiled with the Debug compiler setting,
  • and, if the runtime is tarted with the above environment variable set.
• The GenerateHotReloadDeltas.targets needs some output file work to run hotreload-delta-gen less
  frequently. Right now it runs every time even if everything is up to date.
  For CI testing we should ideally compile the deltas ahead of time once.

To try it out locally:

1. checkout and build hotreload-utils and do dotnet publish --self-contained -r <RID> to
   put hotreload-delta-gen into that projects' artifacts/... forlder. Add the
   publish folder to your $PATH

2. In dotnet/runtime run

DOTNET_MODIFIABLE_ASSEMBLIES=debug MONO_ENV_OPTIONS=--interp ./dotnet.sh build src/libraries/System.Runtime.Loader/tests /p:MonoMetadataUpdate=true /t:Test

(CoreCLR doesn't need MONO_ENV_OPTIONS or MonoMetadataUpdate)

Author:	lambdageek
Assignees:	-
Labels:	area-Infrastructure
Milestone:	-

[mikem8361]

One thing to note that "source build" will require the hotreload-utils repo to built as part of it. I'm not sure exactly how tools like this are handled as part of the source build.

[lambdageek]

One thing to note that "source build" will require the hotreload-utils repo to built as part of it. I'm not sure exactly how tools like this are handled as part of the source build.

Good point. Not sure. I think the situation will be similar to how our LLVM fork, or the other transport nugets are built. (Possibly with a bit more of a bootstrapping problem since hotreload-utils depends on Roslyn which depends on the runtime)

[lambdageek]

various Libraries builds are failing with

Run "dotnet tool restore" to make the "hotreload-delta-gen" command available.

@ViktorHofer @akoeplinger is it not enough to add the reference to eng/Version.Details.xml and to .config/dotnet-tools.json? Is there something else I need to tweak?

[ViktorHofer]

LGTM modulo style nits