VSTest@2: Unable to test .NET 6, vstest.console not found

[mikeKuester]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: VsTestV2

Environment

• Azure Pipelines

• Agent - Hosted: vmImage: 'windows-2022'

Issue Description

I want to upgrade my .NET 5 solution to .NET 6. After every thing works local (VS 2022 Enterprise) I tried to run our Azure Pipeline which builds, tests and publishes the application. After switching the host from "windows-latest" to "windows-2022" the solution could be build with the pipeline, but the tests could not be executed:

... unable to find vstest.console from a visual studio installation ...

I just found a commit "VsTest task changes to support 2022", but it seems, that this update is no available in the pipelines yet?

- task: VSTest@2
  displayName: 'Run tests & publish results"'
  inputs:
    testRunTitle: 'Hardware Independent'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    testAssemblyVer2: '$(testDll)'
    codeCoverageEnabled: true
    runInParallel: true
    rerunFailedTests: true
    rerunMaxAttempts: 3
    testFiltercriteria: 'TestCategory~hardware independent'

Task logs

Starting: Run tests & publish results"
==============================================================================
Task         : Visual Studio Test
Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
Version      : 2.170.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest
==============================================================================
SystemVssConnection exists true
SystemVssConnection exists true
Running tests using vstest.console.exe runner.
======================================================
Test selector : Test assemblies
Test filter criteria : TestCategory~hardware independent
Search folder : D:\a\1\s
Action when minimum tests threshold not met : donothing
Minimum tests expected to be run: 0
VisualStudio version selected for test execution : 16.0
Attempting to find vstest.console from a visual studio installation with version [15.0,17.0).
Attempting to find vstest.console from a visual studio build tools installation with version [15.0,17.0).
##[error]Error: Visual Studio NaN is not found. Try again with a version that exists on your build agent machine.
Finishing: Run tests & publish results"

[aarondandy]

I think I have the same issue but my error is slightly different as I have a custom image with only VS 2022 build tools on it. I think we just need to wait for 2.195.0 to be published.

==============================================================================
Task         : Visual Studio Test
Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
Version      : 2.170.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest
==============================================================================
SystemVssConnection exists true
SystemVssConnection exists true
Running tests using vstest.console.exe runner.
======================================================
Test selector : Test assemblies
Test filter criteria : Category!=SlowTest
Search folder : C:\a\1\s
Action when minimum tests threshold not met : donothing
Minimum tests expected to be run: 0
VisualStudio version selected for test execution : latest
Attempting to find vstest.console from a visual studio installation with version [16.0,17.0).
Attempting to find vstest.console from a visual studio build tools installation with version [16.0,17.0).
Attempting to find vstest.console from a visual studio installation with version [15.0,16.0).
Attempting to find vstest.console from a visual studio build tools installation with version [15.0,16.0).
##[error]Error: Visual Studio 2015 is not found. Try again with a version that exists on your build agent machine.

[aarondandy]

As a workaround for the delayed publish I added 2019 build tools to my custom image. Not ideal, but it unblocked us.

[MatthewSteeples]

Typical timing, but the issue looks to be resolved today
Current image version: '20211109.2'

[MrImpossibru]

@MatthewSteeples wasn't resolved. Version of the image is 20211109.2 but version of the task is still Version : 2.170.1

[MatthewSteeples]

Interesting... Not sure what's different about our configurations then but our VSTest task now runs to completion (it used to give the same error)

[danielheddelin]

Have the same issue. windows-2022.

[Sergio0694]

We're also trying to migrate our pipeline for the Microsoft Store to VS2022, and currently waiting for the VSTest@2 update that has been merged to rollout (can't run any tests otherwise, and we'd like not to also install the VS2019 tools to work around this). Is there an ETA for when the new version will be rolled out, and also how will the fix be pushed to DevOps? Thanks! 😄

[aarondandy]

I have heard from peers with past experience that it can take over 2 months for a merged change to reach us users and it takes 1 hour for me to make a new build image, so I applied the workaround 🤣. Maybe @phanikmmsft, @ShreyasRmsft, @v-pkoneti, @nadesu, or @v-vnomula could provide some insight or set some expectations on the publish date for us.

[kindlaj]

With 2.195.0 task version, it's resolved. Thanks!

[aarondandy]

Our builds are still getting task version 2.170.1 and the task fails if I try to specify 2.195.0 . @ejszi , how did you get the new version of the task to run?

[Sergio0694]

I'm a bit confused about this as well, I tried again the other day and I still got the previous version, which failed.
@ejszi how did you get the task to use the new version for you? 🤔

[kindlaj]

I didn't do anything. The vstest task version just changed:

2021-12-01T08:00:19.5451609Z ##[section]Starting: Test Assemblies
2021-12-01T08:00:19.5633250Z ==============================================================================
2021-12-01T08:00:19.5633525Z Task         : Visual Studio Test
2021-12-01T08:00:19.5634038Z Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2021-12-01T08:00:19.5634654Z Version      : 2.195.0
2021-12-01T08:00:19.5634936Z Author       : Microsoft Corporation
2021-12-01T08:00:19.5636235Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest
2021-12-01T08:00:19.5636810Z ==============================================================================

But now, I have other problem:

Test run detected DLL(s) which were built for different framework and platform versions. Following DLL(s) do not match current settings, which are .NETFramework,Version=v4.0 framework and X86 platform.
......Test.dll is built for Framework .NETCoreApp,Version=v6.0 and Platform AnyCPU.

Obviously, there aren't any netframework x86 dll in my test project.
🥲

[Sergio0694]

I just kicked a new pipeline run again and we're still getting 2.170.1.
Our task is just requiring VSTest@2, so not sure why we seem to be getting a different version that you are 🤔

[tiefling]

I am seeing the same error as of several days ago with .Net Framework projects - VSTest just stopped working.

My builds were updated a while ago (on MS insistence) to windows-latest so I assume they just ticked over to being windows-2022 [I can confirm they ARE showing now as Windows Server 2022].

This is pretty critical stuff. Our deployment pipeline just collapsed.

[tiefling]

I rectified my issue by removing vsTestVersion: "16.0" from the YAML definition.

I can't find a newer version listed in the docs yet but I can only assume there is a new "latest" and removing this just uses that.

[rcmosher]

I just had this issue show up on VsTest version 2.198.0. So a more recent VsTest version doesn't seem to solve it. I just migrated a testing project to .NET 6 and changed the pipeline to use Windows 2022 to fix not having .NET 6 available.

[rcmosher]

I was able to resolve this by changing the Test platform version to latest in my VsTest task. It worked despite the claims this would use VS 2017 or 2015.

[omghb]

I have the same issue as @Sergio0694.

We are using Azure DevOps Server (On Premise) with the latest version. But it still comes with VSTest@2 Task Version 2.170.1. This version does not support VS 2022.

Unfortunately, there is no official release date announced for the next version of Azure DevOps Server.

Right now this blocks us migrating to .NET 6 / VS 2022.

[Sergio0694]

@omghb if it helps, we managed to switch to VS2022 a few months ago using the same workaround @aarondandy suggested: manually also including the VS2019 tools in the VM image used by the pipeline. Not idea, but at least it let us upgrade 😄

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days