Testing.Platform should not be a blackbox: leveraging standard Hosting model and Program.cs for test application setup

[julealgon]

Summary

I've started playing around with the new platform and I'm disappointed in how cryptic and "magical" everything is. Everything is either controlled by random and super undiscoverable MSBuild properties or worse, through environment variables.

With modern applications, the C# ecosystem has shifted from cryptic project-based settings and configuration files, to a much more visible and flexible Program.cs-based C# setup for libraries, settings etc, and this new Testing.Platform should tap into that.

Background and Motivation

The way Testing.Platform works today is just... cryptic, and not idiomatic C# compared to other types of applications. Things are not discoverable, and the way they are setup is through nonstandard means such as csproj flags.

I also want to be able to leverage common patterns and practices that I already leverage in other project types, such as proper dependency injection support, I want to add OpenTelemetry to my tests to monitor their execution in Datadog, I want to add tests from different projects and combine them in a single host, I want to be able to configure details of my tests using our company-wide Azure AppConfiguration instance.

Proposed Feature

I propose that all test projects using the new Testing.Platform have their Program.cs files fully exposed and NOT autogenerated based on .csproj settings. All (or most of at least) MSBuild properties should be deprecated in favor of explicit configuration properties that the developer can set as part of some AddTestingPlatform or UseTestingPlatform call in their Program.cs files, like other types of applications do today.

These methods would be integrated via a new TestApplication.CreateBuilder(args) method that is the entry point to the hosting model used by every other modern application, including ASP.NET (WebApplication.CreateBuilder(args);), Worker projects (Host.CreateApplicationBuilder(args)), Azure Functions (FunctionsApplication.CreateBuilder(args)), etc.

This would then automatically bring first-class support for:

• logging
• metrics
• dependency injection
• configuration

For test projects.

All extension points provided by Testing.Platform would then be DI-friendly, and the hooks would be exposed and configurable in Program.cs, again like any other standard application.

Need to add MSTest support? Add the MSTest nuget package, then call AddMSTest(Action<MSTestOptions>) or some similar method (instead of flipping some weird MSBuild variable. Same for any other test engine.

Need to disable platform telemetry? Set some Option property to false, or bind it to a configuration file or to Azure AppConfig where you set it to false, instead of.... setting an environment variable:

• https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-telemetry#disable-telemetry-reporting

To disable telemetry, set either TESTINGPLATFORM_TELEMETRY_OPTOUT or DOTNET_CLI_TELEMETRY_OPTOUT environment variable to 1.

If this is standardized enough, I could even setup some tests to run as part of my ASP.NET Core API, or I could setup a windows service that runs tests, the same way I can host an API in a windows service today because each of these "aspects" is modular and independent enough and can be combined. The same way I can add ASPNET.Core capabilities to my Azure Function isolated function and have it use common ASP.NET types for HTTP.

Alternative Designs

None. I think any alternative to the above deviates from how modern projects are written and should be avoided.

[Evangelink]

Hi @julealgon,

This was the original intent of our design but to make it easier for users to migrate from the magic VSTest model we had to make compromises.

Note that you can already just disabled all the auto-generated stuff and go with a manual entry point (Program.cs).

I think there is also an implicit question about why we are not using the same ILogger and other interfaces of .NET extensions and ASP.NET and there the reason is both simple and relatively complex. Our experience maintaining VSTest as shown us that we need to be able to test in many scenario including things like runtime where the types available are really limited or in situations where the deps of the testing lib are conflicting with the libs of your app (e.g. VSTest was depending upon Newtonsoft JSON 9 which was conflicting with many user setup)... So we had to make the choice of "copying" some models.

Hopefully, it should be relatively easy to add some extensions that would be bridge to available extensions.

[julealgon]

@Evangelink

This was the original intent of our design but to make it easier for users to migrate from the magic VSTest model we had to make compromises.

This is unfortunate to hear. I was hoping this could be a full restart on how tests are done and that it would be an acceptable breaking change to force people to have to potentially recreate their test projects using the new style (assuming they couldn't figure out how to just adapt their existing ones).

I hope you find a way to evolve this cleanly without all the magic moving forward.

Note that you can already just disabled all the auto-generated stuff and go with a manual entry point (Program.cs).

Oh ok..... that interests me. Can you share a bit more detail around how one approaches this? Is it documented somewhere that you could share a link?

Since the Program.cs automatic generation was mentioned in the .NET Conf video but not explored at all, I figured it wasn't meant to be accessible.

I definitely want to give that a shot.

I think there is also an implicit question about why we are not using the same ILogger and other interfaces of .NET extensions and ASP.NET ...

For sure, yeah.

... and there the reason is both simple and relatively complex. Our experience maintaining VSTest as shown us that we need to be able to test in many scenario including things like runtime where the types available are really limited or in situations where the deps of the testing lib are conflicting with the libs of your app (e.g. VSTest was depending upon Newtonsoft JSON 9 which was conflicting with many user setup)... So we had to make the choice of "copying" some models.

I understand this concern. I saw the Newtonsoft situation mentioned in the .NET Conf video. However I think there is an opportunity here to separate what is the "core" of the engine, from the hosting facilities.

If you check with the OpenTelemetry team, they ended up doing exactly that. OTEL has this "core SDK" which is just the raw stuff, full of static entry points, classes, not much abstraction, etc. And then they have the OpenTelemetry.Extensions.Hosting package, which is the bridge between that low-level core, and all the modern .NET hosting abstractions and systems.

If you could figure how to do a similar split for Testing.Platform, people testing those low-level libraries could use just the core components to reduce dependencies to the bare minimum, while the vast majority of users could leverage the higher abstraction and flexibility provided by the Hosting extension package and get a consistent experience compared to their other project types.

I think I might have mentioned the OTEL example in some other comment a few months ago (I think it might've been in a reply to you too). Perhaps you could reach out to someone on the Microsoft OTEL dotnet side such as @cijothomas or @CodeBlanch and you guys could discuss how to apply some of the principles used by OTEL package design to the Testing Platform as well. It is pretty evident to me that you guys share very similar concerns and problems, and I believe OTEL has already solved a lot of them on their side.

[Evangelink]

Oh ok..... that interests me. Can you share a bit more detail around how one approaches this? Is it documented somewhere that you could share a link?

Since the Program.cs automatic generation was mentioned in the .NET Conf video but not explored at all, I figured it wasn't meant to be accessible.

I definitely want to give that a shot.

This is the sample code to help understand how to extend the various part of the platform https://github.com/microsoft/testfx/tree/main/samples/public/TestingPlatformExamples/TestingPlatformExplorer and it's made as an example that would go with our extensive doc https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-architecture. Reading it again, I am not super satisfied with the shape of it but that was a HUGE chunk of work from @MarcoRossignoli and I to produce it.

I have also added a simpler example for MSTest: #4201