Microsoft.Testing.Platform for xUnit #3478
Comments
Let's start to describe what's inside the
In this case we didn't do any update to the sdk we simply rely on the current support. The task in this case will run the application as-is connecting through a pipe with a custom internal protocol. It means that we're skipping the VSTest infra. NB. We're working on a new version for
The target is defined here https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.targets#L71 The testing framework supports extension and as you know they're linked at build-time. So this task takes also a specific Items collection and generates a call to an extension provided method passing the application builder and the entry point args. So to "plug" the "compile time" registration (vs manual registration where user can manually write the code inside the Main) an extension MUST provide a prop with the registration method. The task will emit the code in the autogenerated main. The sample schema for the Items is for instance the build-time registration of the Trx extension https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Extensions.TrxReport/buildMultiTargeting/Microsoft.Testing.Extensions.TrxReport.props Thanks to the == Why you see Because the So From versioning perspective
//------------------------------------------------------------------------------
// <auto-generated>
// This code was generated by Microsoft.Testing.Platform.MSBuild
// </auto-generated>
//------------------------------------------------------------------------------
[global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
internal sealed class TestingPlatformEntryPoint
{
public static async global::System.Threading.Tasks.Task<int> Main(string[] args)
{
global::Microsoft.Testing.Platform.Builder.ITestApplicationBuilder builder = await global::Microsoft.Testing.Platform.Builder.TestApplication.CreateBuilderAsync(args);
Microsoft.Testing.Platform.MSBuild.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.Testing.Extensions.Telemetry.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.VisualStudio.TestTools.UnitTesting.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.Testing.Extensions.CodeCoverage.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.Testing.Extensions.TrxReport.TestingPlatformBuilderHook.AddExtensions(builder, args);
using (global::Microsoft.Testing.Platform.Builder.ITestApplication app = await builder.BuildAsync())
{
return await app.RunAsync();
}
}
}Collecting from MSBuild 
So I think that if you want to plug to the new testing platform you have to opt-out your one and give us the control on the starting point or extensions won't work. Another possible issue I see is that the new testing platform support custom command line arguments so if you want your users use your v3 options at command line you should provide the extension for it. Here the documentation https://github.com/microsoft/testfx/blob/main/docs/testingplatform/icommandlineoptionsprovider.md with a sample here https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Source/TestingPlatformExplorer/TestingFramework/TestingFramework.CommandLineOptions.cs |
Finally if you plug to the You could have an msbuild property like Another suggestion maybe for next loop is to have all this inside an MSBuild sdk. It helps a lot with clean opt-in/out without giving to users a way to break the config https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-sdk source here https://github.com/microsoft/testfx/tree/main/src/Package/MSTest.Sdk |
When users are asking us to support Another way to phrase this is: if any unit test framework (ignore xUnit.net v3 the moment) only supported If that's the intended goal, to be able to run directly run a project and have it be faster than I personally believe that from a user perspective they're going to expect that we use
We're ready now. 😄 Our current split plan right now with
You've botched the dependency tree, though. We can't unilaterally give you control of our entry point, though. That's out of the question, because our users need to be able to run tests outside of
The two questions I have here aren't around the core of what we're doing right now in that code snippet above, but:
You speak above about the zero-dependency
Digging into those, you can see more upstream dependencies of things that unit test users very well may be using, and therefore have potential places to conflict when different versions are needed that end up not being compatible. Bottom LineIs the "we must own your entry point" a 100% requirement that you are immovable from? If so, then it sounds like we will never support We'd prefer a model where we work with you to find a way so that you can evolve If the entry point ownership is 100% no compromise, then please say that plainly so that I can close this issue and point users here when they ask why we won't support |
We're towards that goal but we added already some benefits that are currently used. First step was have the new platform with all the needed extensibility point. So the
I saw our branch, happy to see that the documentation and the sample helped to quickly integrate. On our side we need a bit more time to "adapt" to the rest of the environment and tooling.
We solved that problem with the MSBuild SDK there you can conditionally opt-in out set of different packages. We switch completely the implementation of the engine when we go AOT. Preserving the UseVSTest for users that wants to opt-in old set of packages.
What you mean?
What's missing?
We ship by default with our console UX to report the result and the errors but we don't capture the output in any way, if you do
This is not super clear as question, at the moment we're doing what you said, we use MSBuild prop Items that we codegen at build-time as I've explained above(https://github.com/microsoft/testfx/blob/main/src/Platform/Microsoft.Testing.Platform.MSBuild/buildMultiTargeting/Microsoft.Testing.Platform.MSBuild.props). Extension will be shipped by us and by users and if they provide the correct Item inside the prop they will register themselves without the need of any change inside the program.cs. Otherwise users can disable the codegen and do it manually with the classic extension method.
For a test adapter the only dependency is
After users can add extensions and extensions can take dependencies our of our control.
You don't need
If you copy our way to at compile time register the dependency you can take it. It's explained above, if you codegen the registration by your own it's fine we don't need the entry point. You need to flow the options to our builder for server mode (for
Our goal is to drop the VSTestBridge so you're one step ahead. If you want to wait till we will have complete the new dotnet test and the TestExplorer to complete the integration I think it's understandable. We need to understand how to handle the command line issue. If you try a |
Your claim was that you auto-generate this entry point (which, BTW, I cannot duplicate): [global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
internal sealed class TestingPlatformEntryPoint
{
public static async global::System.Threading.Tasks.Task<int> Main(string[] args)
{
global::Microsoft.Testing.Platform.Builder.ITestApplicationBuilder builder = await global::Microsoft.Testing.Platform.Builder.TestApplication.CreateBuilderAsync(args);
Microsoft.Testing.Platform.MSBuild.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.Testing.Extensions.Telemetry.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.VisualStudio.TestTools.UnitTesting.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.Testing.Extensions.CodeCoverage.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.Testing.Extensions.TrxReport.TestingPlatformBuilderHook.AddExtensions(builder, args);
using (global::Microsoft.Testing.Platform.Builder.ITestApplication app = await builder.BuildAsync())
{
return await app.RunAsync();
}
}
}My counterpoint was that several of these lines would not compile without additional references; specifically: Microsoft.Testing.Extensions.Telemetry.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.VisualStudio.TestTools.UnitTesting.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.Testing.Extensions.CodeCoverage.TestingPlatformBuilderHook.AddExtensions(builder, args);
Microsoft.Testing.Extensions.TrxReport.TestingPlatformBuilderHook.AddExtensions(builder, args);Given that I tried to repro this, and saw only this as your auto-generated entry point, I'm confused by what you claimed previously:
Note that this is exactly what we're doing as well:
(The extra Register call is us registering ourselves as a test framework.) So I'm not sure what/how we go from what I'm seeing today to what you claimed above. Is your claim above an idealized goal for the future? Is the auto-generation dependent on the user adding references to NuGet packages that include register-able things (for lack of a better term)? Are you sniffing the
I'll take your word for it if you say that you don't ever output anything to the console. My experience so far has been with
Does this mean I only have to look for you to pass
If there's value in me providing something in the meantime that people could opt into that would work, then I'd be happy to do that (as I presume it would also help you all do testing and validation of the proposed usage). Today that looks like me including Also today, as #3479 illustrates, I have no idea whether I'm being used via TPLv2 or I feel like, based on this conversation, the appropriate path is probably:
Given the non-intrusive nature of this (basically, I'm just sniffing for |
|
I do see one problem right now with my current thought: <TestingPlatformBuilderHook Include="98058041-B5B6-4A75-9834-58E6DF796A22" Condition=" '$(GenerateTestingPlatformEntryPoint)' == 'True' " >We obviously have to set Is there any reason this should actually be conditional? What's the harm in all of your extensions just always having (That said, I could make it work today, because removing |
|
I have updated the If you want to take a look at the changeset and see if this aligns with what you'd expect: xunit/xunit@0073f0b This is a project that just references
It references the // <auto-generated> This file has been auto generated by xunit.v3.core. </auto-generated>
[global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
public class XunitAutoGeneratedEntryPoint
{
public static int Main(string[] args)
{
if (args.Length > 0 && args[0] == "--server")
return global::Xunit.Runner.InProc.SystemConsole.TestingPlatform.TestPlatformTestFramework.RunAsync(args, typeof(Microsoft.Testing.Extensions.TrxReport.TestingPlatformBuilderHook)).GetAwaiter().GetResult();
else
return global::Xunit.Runner.InProc.SystemConsole.ConsoleRunner.Run(args).GetAwaiter().GetResult();
}
}I followed the pattern of the code you linked to that ensures that the TrxReport extension is always registered last. |
I feel like maybe an optional priority might be better than hard coding this logic. |
Yes we're "sniffing the TestingPlatformBuilderHook" so the lines you see inside the codegen depends on the extension packages added.
I agree, we discussed and for first version we decided to embed it. But it's on the table.
I think that is not fundamental have 100% same UX for the help. Under the new(in-progress) "dotnet test --help" for instance we aggregate all the options and we notify the users if some tests module has got different options (usually brought by extensions) because at runtime for that specific test module with the missing extension if will fail for invalid option. I mean the UX will be for sure different. So I think that you can map all your options under the testing plat and for the option without a "-some" add a new one i.e. --config ... --seed ... only when is running in testing platform mode. Despite your UX looks really well organized I think that at least for the start have a plain list where users can read and maybe look at the documentation if something is not clear is a good starting point. Usually once users understand the option they won't change it for some time. I mean the important thing I see is that users can in some way opt-in/out all the xunit feature. PS. It's not so bad idea give an extensibility point where test framework author can write the custom help, I'll open a ticket for it. cc: @Evangelink
In your ITestFramework implementation you can get thought the IServiceProvider the ICommandLineOptions and query the command line and apply the options. For instance https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Source/TestingPlatformExplorer/TestingFramework/TestingFramework.cs#L47
The ICommandLineOptionsProvider allows you to validate single options and global configuration at the end for instance https://github.com/microsoft/testfx/blob/main/docs/testingplatform/Source/TestingPlatformExplorer/TestingFramework/TestingFramework.CommandLineOptions.cs
At the moment we support
We give a way to personalize the banner implementing the IBannerMessageOwnerCapability and at the moment we ship our built-in output console. But we can if you need add an option that will allow you to have headless runtime and you can write your own UX. If it's a requirement for you open a ticket for it.
Yes, in
True current dotnet integration that rely on the current(old) MSBuild infra can be recognized using
VS decide if start using the server mode (start process passing args) looking at the
I think you could keep both deps
The reason is to not pollute MSBuild with unused items. We didn't expect the needs to provide the same items outside our codege. But it's fine for me remove the conditional and let you sniff.
I'll check and let you know soon. Thanks for the effort. |
|
I was wondering if to solve the issue with the codegen for the extension we can do something simpler where you can keep the Microsoft.Testing.Platform.TestingPlatformBuilderHook.AddExtensions(builder,args);So you can simply emit in your code and we take care of the dirty sniff. And we update our codegen to call the same method for symmetry. |
I'm not talking about what the user sees with At the moment I think of
Agreed. The interesting problem with A related question: how will these command line options be surfaced into Test Explorer, if at all? To use the seed example, would I be able to set a seed for a specific test assembly (at a specific target framework) with some UI gesture in Test Explorer?
For the moment, since we're trying to achieve the ability for me to still own my entry point, I don't think we need to open any other issues. I'd personally prefer for me to continue to own my entry point, at which point it becomes unnecessary for me to own the banner or anything else like that (since it would only be seen in the case of the user running
This is a simple change for me to make (rather than assuming it's first).
Personally I'm fine with I'm also fine if it turns out that some other test framework wants this hidden option, we can look for that hidden option instead.
You should be able to test this with my branch right now. All I do is run I also use a function that I wrote that cleans out function Clean-DevPackages {
$packageCache = $ENV:NUGET_PACKAGES
if ($null -eq $packageCache) {
$packageCache = [System.IO.Path]::Combine($home, ".nuget", "packages")
}
if ($null -ne $packageCache) {
Get-ChildItem ([System.IO.Path]::Combine($packageCache, "xunit.*")) | ForEach-Object {
Get-ChildItem ([System.IO.Path]::Combine($_.FullName, "*-dev")) | ForEach-Object {
Write-Host $_.FullName -ForegroundColor Yellow
Remove-Item -Recurse -Force $_
}
}
}
}Alternatively, if you can point me to instructions on enabling this preview Test Explorer feature (assuming I can do so), I could test this integration myself.
I agree that shipping what I have right now to end users doesn't buy them anything, because (a) there's no I'd be willing to ship this as-is to help your team use us as a second implementation (that does not use VSTestBridge) to verify that everything has been documented and working from the core protocol/API perspective. Obviously it requires either implementing my or your suggestion on how to work around my problems today with needing
Part of the end user documentation of this experimental feature could be to not only say "you need to turn on I could give them a simple copy/paste block to add those three defaults, which is a sneaky way to ensure they were using a version that's compatible (if we end up needing versions without the conditional on Later on, when this becomes non-experimental, it would make sense for us to update our project templates for v3 to include these extensions by default (much like the v2 templates today include coverlet, as a way to document "here's how you enable this" in a way that users can opt out of by simply deleting the dependencies they don't care about). How permanent is
I think is is better than my suggestion, because it leaves the ownership of the discovery and ordering (and whatever else you might need in the future) to you, rather than duplicated by me. You can auto-generate this static method in a well known namespace & class, and I could push a reference to it (as a lambda) into my existing compiled |
|
I've updated my branch, and with this update, I can use
If you don't set |
We can add the alias --? to our --help I think it's fine. #3501
Ok I think make sense.
True, we forward all param to all underneath test modules, if you output the seed on console with the output users can repro or am I out of track?
At the moment we have fixed way to startup the modules in Test Explorer, but we discussed the concept of profile file (TBD if use VS or VS Code or custom launcher format) where users can create a profile file in the project self and decide which one to run, but this needs some UX updates. cc: @drognanar
I wonder if you want to support also our runner without
Ok let's start with
I don't think it's shipped in public yet so I need to test it by my own, @drognanar can you confirm that's only internal for now?
I did it quickly yesterday the clone I got some sec issue with 9 preview so I need to add a
That's great help for use you're not using the bridge so we can fold the difference at protocol level inside the Test Explorer(VSTestBridge is adding some custom
You can point that part of the docs to our extension list where there's the description and the setup https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-extensions in that case you can avoid to add any extension by default users can start adding manually in their csproj. After you can decide if implement an SDK(imo great clarity for users, they see their csproj completely empty and with MSBuild knobs you can really change everything without any confusion on what packages needs to be added or remove or needs of complex porting documentation, but it's my opinion).
We will remove the conditional and offer the extension registration you can track this label https://github.com/microsoft/testfx/issues?q=is%3Aissue+is%3Aopen+label%3Axunit
Up to you
I'd says that they will coexist for at least 3/6(worst case) months after our
We have discussed with @Evangelink today about it #3494 |
I think that the idea to drop VSBridge/VSTest for v3 will be also useful for us to understand if something is missing to "deprecate" VSTest for .NET. We don't have the luxury to drop it with MSTest yet and xunit is great help to understand what we really need to deprecate the glorious but a bit tired VSTest. |
Yep, this is a good idea. Right now our only way to "push" into your command line parser is by picking up on In order to implement this and make it usable, we'll also need to make sure we register our command line options (we don't do that yet).
No, I generally try to keep my machine free of preview builds, and I also try not to be overly prescriptive with Since our minimum platform right now is .NET 6, the only reason we'd push to .NET SDK 8 is to take advantage of newer compiler features. I'll fire up a Linux machine with a .NET SDK 9 preview and see what the breaking changes are. |
A cleaner idea could be also to allow users to specify in their project So they can opt-in it globally using directory props. |
|
Done: xunit/xunit@b7806ed
|
|
I did a logic flip: instead of checking for
|
Great, one detail that should not affect the entry point but I share so we can co-think about it, the testing platform is written to be Single module deploy it means that to support single file/aot with extensions we cannot have a build with "more than one file" in these formats. This means that for instance if a user register a native aot out of process extension we "restart" ourself passing the original parameter plus some more parameter to handle the information between the the test host controller(out of process that can observe the process that hosts the run of the tests) and the test host(where the adapter run the tests). I think that this should not affect anything anyway you will forward to the correct branch due to the fact that we don't "add" -automated or @@. I wonder anyway if in future it's possible that someone will ship some extension with |
|
FYI you can build against our dev feed till we will reach a good integration so we can verify the progress https://github.com/microsoft/testfx/blob/main/docs/preview.md |
At the moment we look for exactly I included I'll have to think about whether there is concern about the
At the moment we are not planning to support AOT. I've had some discussions about what would be required to make this happen but it is not anywhere near the top of the backlog. |
Ok let's sleep on it
Ack |
I've pushed up setting these properties. You should be able to validate it on your end now with your fixed Test Explorer. Just add a test that calls |
|
I should be done with all the tests now, so unless we track down any more issues, I think I'm basically just waiting for the 1.4 release later this month. 😄 I'm assuming the VS 2022 17.12 final release will probably coincide with .NET 9 in November. |
That's great, I think we're done here for now, we will release the 1.4 soon cc: @Evangelink
This is TBD we didn't complete the VS side work yet we're close but not yet there so it's also possible that we will have another preview round for 17.13 and have it as "public preview toggle" in 17.13 GA. As you can image a lot of users are using the test explorer so we need to progress with caution. cc: @pavelhorak Thanks for your effort here @bradwilson and feel free to suggest feature or improvement if you something! |
|
I can't think of anything for the Platform for the moment. We already have the other issue discussing the "Not Run" state, but I think from a UX perspective in Test Explorer we're in the same place we were before and a custom state is not necessary there. It would be nice to be able to alert the user to the "not run" test in the summary, but that's still a fairly low priority, IMO (and if xUnit.net users care about that, they can use our CLI UX and/or our reports for it, since we'll put it in the report even if the MTP CLI UX doesn't). I was going to ask whether you thought I should enable |
|
From a release schedule perspective, do you know when you plan to release the newer/final version of |
|
I haven't tested, but in theory couldn't you just push a new test node with the DiscoveredState on it? Might cause the test explorer to replace in progress with not run? |
|
To be clear: Test Explorer is behaving exactly how I want it to behave now. I just want to make sure I'm not breaking anything by telling the Platform that I started running a test but never telling it that it finished. |
|
Also, I'd find it surprising that the Platform wouldn't find the behavior of giving a discovery result during an execution request more unexpected than just starting but never finishing a test. 😂 |
|
I've fixed it with 3.0.0-pre.31. |
Yep for this one let's use the other issue it's a Text Explorer use case unrelated to the testing platform, despite we could add a new state if the UX cannot handle it it's useless at the moment.
We will add the opt-in flag in the preview box...so the idea is if the flag is opted-in plus you expose TestingPlatformServer we go for it. cc: @drognanar if my expectation is wrong.
Yep, we call it "MSBuild VSTest task workflow"(the name of the built-in target to plug to). We don't have ETA yet about the release of the new
@thomhurst if we want to add this new "state" and we think it's something that make broadly sense for adapters as a known shared scenario I prefer to have a specific state for it. We should not base states on Test Explorer, if we need we will go amend there the recognition of this state.
At the moment "the platform" is nothing more than a Ticket opened for discussion for the next iteration #3759
@drognanar can you help here? Is it a problem for the current Test Explorer expectations? |
I've opened #3699 around an Inconclusive state. There's discussions around it being similar to Skipped, but I see Skipped as a user decision. But keen to hear other people's thoughts. |
|
I'm going to close this issue because we're done with the "first integration wave" for xunit and this ticket is needed to complete the 1.4 milestone. cc: @Evangelink @nohwnd |
|
FYI, I'm adding |
|
@bradwilson Thank you for the nice collaboration! Looking forward for your alpha/beta/preview and for the next steps of our collaboration. I was off for a week, I am catching up with notifications and will be reading/replying to all points as I move through them. |
|
@bradwilson we've shipped official |
|
My branch is merged, and the build is available as I've created a documentation page: https://xunit.net/docs/getting-started/v3/microsoft-testing-platform The documentation page is not yet linked from the home page, because I wanted to give your team a chance to review it regarding language/links/etc. and see if there are any changes you'd like to see. You don't have to be concerned with the xUnit.net technical details; you can just assume I got them right. 😂 But obviously if I got any of the technical details about MTP wrong, please let me know! Once the documentation is good, then you can update https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-intro with a link to the doc page. I won't make a full announcement until I ship the next prerelease build to NuGet. I probably won't do that until I finish the filter query stuff. I'm waiting on feedback from @Evangelink here before I finalize my implementation decisions, as I'd like to try to match the MSTest implementation as close as is reasonable. |
|
I'll read the doc and I'll give it a try and after I'll link to our docs.
|
|
Same here, I've read the docs, and checked most of the links and all looks good. Great job :) |
|
So I've read and did all the sample and it's all good and working as expected, one nit is the output of the new Anyway I think it's not so important, it's possible that we will update at every version till steady state so I think it can stay as-is. I'll link the doc to our docs. PR out dotnet/docs#42614 |
|
Yeah, the difference is .NET SDK 8 vs. 9. I opted to include the output from 8 since 9 is still prerelease. |
|
Sounds like the docs are good to go, so I just pushed up the link from the home page. If you have any feedback, feel free to leave it here (or you can submit a PR to the |
|
Thanks! |
|
FYI, we have just officially shipped |
|
Amazing thanks! |
The approach xUnit.net v3 is taking for supporting Microsoft.Testing.Platform (since it already produces executables) is to conditionally use the "create
TestApplication.CreateBuilderAsyncand build it" approach when it understands that it's time to run in that way, vs. letting it run with our entry point.Our current strategy is to:
Microsoft.Testing.PlatformandMicrosoft.Testing.Platform.MSBuildxunit.v3.coreNuGet package to indicate that a v3 test project is in placexunit.v3.runner.testingplatform) inside ourxunit.v3.coreNuGet package that has all theMicrosoft.Testing.Platformlogicxunit.runner.visualstudiowhen it's got a v3 test project, and override the entry point to conditionally dispatch to the code insidexunit.v3.runner.testingplatformwhen it realizes that we're being asked to run inside ofdotnet testThere are two branches right now that perform this work in concert with one another:
https://github.com/xunit/xunit/tree/microsoft-testing-platform
https://github.com/xunit/visualstudio.xunit/tree/microsoft-testing-platform
My question is about the documentation & support behind the hook. Right now, through a combination of decompilation of
Microsoft.Testing.Extensions.VSTestBridge(which we are not using) and inspecting the logs fromdotnet testwhen using that, it appears thatdotnet testis utilizing a command line option that looks like--internal-msbuild-node <argument>.Based on this comment, this is currently an undocumented and unsupported way to get
dotnet testto work:So my question is:
When will the interaction between Microsoft.Testing.Platform test frameworks and
dotnet testbecome documented and supported? If this isn't the way we should be doing it, then what is the officially supported way to enabledotnet testfor Microsoft.Testing.Platform test framework implementations that are not based onMicrosoft.Testing.Extensions.VSTestBridge?The text was updated successfully, but these errors were encountered: