xunit.runner.visualstudio 2.8.1 fails to discover .Net 4.7.2 Unit Tests

[onur-ozguzel]

Hello,

For a long time, we were using xunit.runner.visualstudio 2.4.5. vstest.console.exe was able to detect both our .NET 6.0 and .NET Framework 4.7.2 unit tests with these parameters:

vstest.console.exe "C:\a\2\s<Full path of first test assembly>"
"C:\a\2\s<Full path of the second test assembly>"
"C:\a\2\s<Full path of the third test assembly>"
/TestCaseFilter:"Category=UnitTest"
/Settings:"C:\a_temp\lbxw3zg1gnt.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\a\2\s"
/Framework:.NETFramework,Version=v4.7.2

After we upgraded to xunit.runner.visualstudio 2.8.1, our .NET Framework unit test discoveries stopped working.

We tested all the new versions, the results were the same except 2.5.0. So discovery were not working correctly for 2.8.1 - 2.8.0 - 2.5.8 - 2.5.7 - 2.5.6 - 2.5.5 - 2.5.4 - 2.5.3 - 2.5.1.

The error message we were getting was:

No test matches the given testcase filter Category=UnitTest in "C:\a\2\s<Full path of first test assembly>"
No test matches the given testcase filter Category=UnitTest in "C:\a\2\s<Full path of second test assembly>"
No test matches the given testcase filter Category=UnitTest in "C:\a\2\s<Full path of third test assembly>"

For discovery to work again for .NET 4.7.2 Unit Tests, we had to change TestAdapterPath like below:

vstest.console.exe "C:\a\2\s<Full path of the first test assembly>"
C:\a\2\s<Full path of the second test assembly>"
C:\a\2\s<Full path of the third test assembly>"
/TestCaseFilter:"Category=UnitTest"
/Settings:"C:\a_temp\qfap23dg5er.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\a\2\s<Full folder path of one of the test assemblies>"
/Framework:.NETFramework,Version=v4.7.2

We have our project in C:\a\2\s folder. Meaning inside this folder, we have both .NET 6.0 and .NET Framework 4.7.2 Unit Tests and Test adapter assemblies. Our hunch is, after xunit.runner.visualstudio 2.4.5 and 2.5.0, vstest.console.exe started to find the wrong test adapter for .NET Framework 4.7.2 tests if we provide the base path, and because of that it failed to discover them.

In the newer versions of xunit.runner.visualstudio, can we provide the base path of the TestAdapterPath and make it work just like it was working in 2.4.5 and 2.5.0?

PS: For .NET 6.0 unit test, it is still able to discover tests by providing the base path to the TestAdapterPath like below:

vstest.console.exe "C:\a\2\s<Full path of first test assembly>"
"C:\a\2\s<Full path of the second test assembly>"
"C:\a\2\s<Full path of the third test assembly>"
/TestCaseFilter:"Category=UnitTest"
/Settings:"C:\a_temp\wizpx4ngr2k.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\a\2\s"
/Framework:.NETCoreApp,Version=v6.0

[bradwilson]

Our hunch is, after xunit.runner.visualstudio 2.4.5 and 2.5.0, vstest.console.exe started to find the wrong test adapter for .NET Framework 4.7.2 tests if we provide the base path, and because of that it failed to discover them.

I suspect your hunch is correct, in which case the bug is in VSTest and not with us, because we don't control that directory-based adapter discovery logic.

I don't see much that changed between 2.5.0 and 2.5.1 that looks "suspect"; it was mostly pulling newer versions of xunit.runner.utility from the core project. If the VSTest wants to try to narrow it down on our side, we offered a few pre-release 2.5.1 packages that might help them.

[MichelZ]

I'm currently looking into a similar issue I think, but with dotnet test.
After updating versions, tests are not discovered anymore for net462:
https://sqlclientdrivers.visualstudio.com/public/_build/results?buildId=101590&view=logs&j=227d917a-c052-5356-3796-5eec3da988b1&t=485a8f67-157b-58a1-dc6b-ec09440e36b5

Here's the source PR with the changes: dotnet/SqlClient#2980

I've tried --test-adapter-path with no luck though.

@onur-ozguzel did you open any issue in the VSTest repo?

[MichelZ]

We were able to resolve this it seems by removing these from the test project(s):

 <PackageReference Include="xunit.runner.console" Version="$(XunitVersion)">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <PackageReference Include="xunit.runner.utility" Version="$(XunitVersion)" />

Thanks to dotnet/runtime#94183 which pointed me into the right direction it seems.

[bradwilson]

This in particular is the problem, due to version conflicts:

<PackageReference Include="xunit.runner.utility" Version="$(XunitVersion)" />

Why do you have this linked?

[MichelZ]

It's from the https://github.com/dotnet/SqlClient repo. I have no idea why this is linked there, I was just trying to update XUnit

[onur-ozguzel]

I'm currently looking into a similar issue I think, but with dotnet test. After updating versions, tests are not discovered anymore for net462: https://sqlclientdrivers.visualstudio.com/public/_build/results?buildId=101590&view=logs&j=227d917a-c052-5356-3796-5eec3da988b1&t=485a8f67-157b-58a1-dc6b-ec09440e36b5

Here's the source PR with the changes: dotnet/SqlClient#2980

I've tried --test-adapter-path with no luck though.

@onur-ozguzel did you open any issue in the VSTest repo?

I didn't create any issue. We've started to consume the way I described above and forget about it. But I am happy you solved your problem.

[bradwilson]

The good news is that xunit.runner.visualstudio version 3.x.y builds are using xunit.v3.runner.utility which supports v1/v2/v3 projects, and we are using ILRepack to package it directly into the test runner assembly, so there will be no more conflicting versions of that. The only other dependency will continue to ship is xunit.abstractions.dll, which has been binary identical since we shipped xUnit.net 2.0.0 in 2015. (There have been four package versions, but the DLL remains the same from a binary perspective.)

    Directory: ~\.nuget\packages\xunit.runner.visualstudio\3.0.0-pre.49\build\net472

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---         2018-Aug-25    12:54          21928 xunit.abstractions.dll
-a---         2024-Nov-04    12:16            705 xunit.runner.visualstudio.props
-a---         2024-Nov-04    12:16        1982160 xunit.runner.visualstudio.testadapter.dll