Move off from VSTest so we can get test console log failures

[joperezr]

Due to a bug in vstest, when a test fails and we are running them in parallel we may lose the console output making it impossible to debug the test failure, especially when we have flaky tests. This issue is to track the work of move away from vstest and use the runner directly to avoid this issue.

[ViktorHofer]

cc @nohwnd

This issue is to track the work of move away from vstest and use the runner directly to avoid this issue.

Which runner do you mean? xunit.console or invoking an assembly directly with dotnet test (and with that still use the VSTest runner)? IMO moving away from VSTest to xunit.console would be unfortunate as the latter is not supported and serviced anymore.

[joperezr]

Doesn't dotnet/runtime invoke the test runner (xunit.console) directly via their generated RunTest scripts?

[ViktorHofer]

Yes, but mostly for historic reasons and because VSTest is too heavy weight for runtime's usecase. xunit.console is no longer maintained by Brad.

[raffaeler]

@ViktorHofer do you have an alternative proposal for our repo?
We have a really hard time with the current host and apparently the solution will not happen in the near future.
Isn't better migrating to the same infrastructure used by the runtime?

[ViktorHofer]

Isn't better migrating to the same infrastructure used by the runtime?

Definitely not. Our infrastructure only exists because of corner case requirements (we build the runtime ourselves) that other repositories shouldn't need to consider. I would recommend to stick using VSTest and work with their team on a path that resolves your issues so that customers would benefit from that as well.

@Evangelink @nohwnd who would be the best person to talk to about microsoft/vstest#2702 to prevent iot moving away from VSTest?

[Evangelink]

Thanks for the ping @ViktorHofer! @joperezr Sending you an invite for a call (will cc you @ViktorHofer).