Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dotnet test is not running tests for .NETFramework targets #94183

Closed
ericstj opened this issue Oct 30, 2023 · 25 comments · Fixed by #96826 or #106721
Closed

dotnet test is not running tests for .NETFramework targets #94183

ericstj opened this issue Oct 30, 2023 · 25 comments · Fixed by #96826 or #106721
Labels
area-Infrastructure-libraries dev-innerloop in-pr There is an active PR which will close this issue when it is merged
Milestone
9.0.0

Comments

@ericstj
Copy link
Member

ericstj commented Oct 30, 2023

Description

When running dotnet test on a test project in dotnet/runtime that targets .NETFramework I expect tests to run, but they do not. Seems like we have lots of other issues mentioning this, but I couldn't find an active one.

Reproduction Steps

build clr+libs -rc Release
cd src\libraries\System.Collections.Immutable\tests
dotnet test -f:net462

Expected behavior

Tests run.

    Discovering: System.Collections.Immutable.Tests (app domain = on [no shadow copy], method display = ClassAndMethod,
   method display options = None)
    Discovered:  System.Collections.Immutable.Tests (found 4008 of 4123 test cases)
    Starting:    System.Collections.Immutable.Tests (parallel test collections = on, max threads = 16)

Actual behavior

No tests run:

Test run for c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll (.NETFramework,Version=v4.6.2)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Regression?

I think this is a recent regression. I believe this used to work as I remember seeing test failures on framework when running with dotnet test.

Known Workarounds

Directly invoke the test target.

\src\dotnet\runtime.dotnet\dotnet build /t:test -f:net462

Configuration

No response

Other information

No response
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Oct 30, 2023
@ghost
Copy link

ghost commented Oct 30, 2023

Tagging subscribers to this area: @dotnet/area-system-collections
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

When running dotnet test on a test project in dotnet/runtime that targets .NETFramework I expect tests to run, but they do not. Seems like we have lots of other issues mentioning this, but I couldn't find an active one.

Reproduction Steps

build clr+libs -rc Release
cd src\libraries\System.Collections.Immutable\tests
dotnet test -f:net462

Expected behavior

Tests run.

    Discovering: System.Collections.Immutable.Tests (app domain = on [no shadow copy], method display = ClassAndMethod,
   method display options = None)
    Discovered:  System.Collections.Immutable.Tests (found 4008 of 4123 test cases)
    Starting:    System.Collections.Immutable.Tests (parallel test collections = on, max threads = 16)

Actual behavior

No tests run:

Test run for c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll (.NETFramework,Version=v4.6.2)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Regression?

I think this is a recent regression. I believe this used to work as I remember seeing test failures on framework when running with dotnet test.

Known Workarounds

Directly invoke the test target.

\src\dotnet\runtime.dotnet\dotnet build /t:test -f:net462

Configuration

No response

Other information

No response

Author: ericstj
Assignees: -
Labels:

area-System.Collections
Milestone: -

@ghost
Copy link

ghost commented Oct 30, 2023

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

When running dotnet test on a test project in dotnet/runtime that targets .NETFramework I expect tests to run, but they do not. Seems like we have lots of other issues mentioning this, but I couldn't find an active one.

Reproduction Steps

build clr+libs -rc Release
cd src\libraries\System.Collections.Immutable\tests
dotnet test -f:net462

Expected behavior

Tests run.

    Discovering: System.Collections.Immutable.Tests (app domain = on [no shadow copy], method display = ClassAndMethod,
   method display options = None)
    Discovered:  System.Collections.Immutable.Tests (found 4008 of 4123 test cases)
    Starting:    System.Collections.Immutable.Tests (parallel test collections = on, max threads = 16)

Actual behavior

No tests run:

Test run for c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll (.NETFramework,Version=v4.6.2)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Regression?

I think this is a recent regression. I believe this used to work as I remember seeing test failures on framework when running with dotnet test.

Known Workarounds

Directly invoke the test target.

\src\dotnet\runtime.dotnet\dotnet build /t:test -f:net462

Configuration

No response

Other information

No response

Author: ericstj
Assignees: -
Labels:

area-Infrastructure-libraries, untriaged
Milestone: -

@ViktorHofer ViktorHofer added dev-innerloop and removed untriaged New issue has not been triaged by the area owner labels Oct 30, 2023
@ViktorHofer ViktorHofer added this to the 9.0.0 milestone Oct 30, 2023
@ViktorHofer
Copy link
Member

Someone needs to debug this. This likely also impacts VS Text Explorer which uses the VSTest runner underneath. There are settings to get more verbose logs returned by VSTest.

@ViktorHofer
Copy link
Member

One more thought, that could be related to the recent xunit 2.5.x upgrade.

@ViktorHofer
Copy link
Member

ViktorHofer commented Oct 30, 2023 •
edited
Loading 


TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.757, 3877444491007, vstest.console.dll, MetadataReaderExtensionsHelper: Discovering extensions inside assembly 'Microsoft.Diagnostics.NETCore.Client, Version=0.2.8.21101, Culture=neutral, PublicKeyToken=31bf3856ad364e35' file path 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions\Microsoft.Diagnostics.NETCore.Client.dll'
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.766, 3877444578798, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Resolving assembly.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.766, 3877444580691, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Searching in: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions'.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.766, 3877444582066, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions\Microsoft.Bcl.AsyncInterfaces.dll', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444583023, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions\Microsoft.Bcl.AsyncInterfaces.exe', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444583316, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Searching in: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1'.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444583907, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Microsoft.Bcl.AsyncInterfaces.dll', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444584450, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Microsoft.Bcl.AsyncInterfaces.exe', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444584752, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Failed to load assembly.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444585113, vstest.console.dll, CurrentDomainAssemblyResolve: Resolving assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444585764, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Resolving assembly.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444586390, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Resolved from cache.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444586674, vstest.console.dll, CurrentDomainAssemblyResolve: Resolving assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444586988, vstest.console.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.768, 3877444596875, vstest.console.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Warning: 0 : 25352, 1, 2023/10/30, 19:39:45.783, 3877444748536, vstest.console.dll, TestPluginDiscoverer: Failed to get types from assembly 'Microsoft.Diagnostics.NETCore.Client, Version=0.2.8.21101, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos, String filePath) in /_/src/Microsoft.TestPlatform.Common/ExtensionFramework/TestPluginDiscoverer.cs:line 158
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
File name: 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
File name: 'Microsoft.Bcl.AsyncInterfaces, Culture=neutral, PublicKeyToken=null'
   at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginCache.CurrentDomainAssemblyResolve(Object sender, AssemblyResolveEventArgs args) in /_/src/Microsoft.TestPlatform.Common/ExtensionFramework/TestPluginCache.cs:line 513
   at System.Runtime.Loader.AssemblyLoadContext.GetFirstResolvedAssemblyFromResolvingEvent(AssemblyName assemblyName)
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(AssemblyName assemblyName)
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr gchManagedAssemblyLoadContext, AssemblyName assemblyName)

--diag:log.txt

@ViktorHofer
Copy link
Member

Filed xunit/visualstudio.xunit#390

@bradwilson
Copy link

This is not what you think it is.

image

@bradwilson
Copy link

Are you're linking against xunit.runner.utility for some reason? If so, then you're frozen to using whatever version of xunit.runner.utility the VS adapter uses (because VSTest demands all dependencies be copied into the output folder). 99%+ of users don't do this so this isn't a general problem.

@bradwilson
Copy link

The only reason to link against xunit.runner.utility is because you're creating your own runner. Tests should never link against this unless they're forced to because they're testing the runner you're writing.

@bradwilson bradwilson mentioned this issue Oct 30, 2023
Closed
@bradwilson
Copy link

bradwilson commented Oct 30, 2023 •
edited
Loading

For a bit of architectural background...

xUnit.net splits what it does into two separate sections which do not directly interact except via very strict contracts. There are "runner" and "execution" halves of the system. In v1/v2 when using .NET Framework, these two halves can be optionally separated by an app domain boundary (and in v2 .NET Core, they live in the same process). In v3 against all target platforms, they're separated by a process boundary.

  • Tests lives in the "execution" half, and are linked against xunit.core and xunit.execution.* (and usually xunit.assert, though that's optional).
  • Runners live in the "runner" half, and are linked against xunit.runner.utility.* (and maybe other runner related things).
  • The only common assembly that lives in both halves, and comprises the communication contract, is xunit.abstractions. It has been frozen since we shipped 2.0 (we have shipped newer versions of the NuGet assembly just to add extra targets as 2.0 was back in PCL259 days, but the DLL itself remains unchanged since we shipped it).

By keeping this strong separation and frozen contracts, we have both backward compatibility (newer runners can run older tests) and major forward compatibility (older v2 runners can run newer v2 tests). In point of fact, I just was able to successfully run tests linked against core framework 2.5.3 with the VSTest adapter 2.0.0, which is 8.5 years old and targeted net20, wpa81, and win8. Obviously .NET Core didn't even exist back then, so it wouldn't be able to run a platform it doesn't know about, but .NET Framework is not a problem.

If your tests link against xunit.runner.utility.*, you're bringing the runner half of things where it normally doesn't live or belong. This is unavoidable if you're writing your own runner, and want to test that runner (we obviously do this, but it should be very very rare). Normally this wouldn't be a problem except when you're using VSTest, because there is a requirement placed by VSTest that we copy the VSTest adapter into your output folder, which of course means we also have to copy the runner utility libraries (xunit.runner.utility.* and xunit.runner.reporters.*). This is the only scenario where we end up with potential collisions between versions of the runner utility libraries, so when you're in this scenario, you must very explicitly remain frozen to use the same version of the runner utility libraries as the VSTest adapter is that you're using.

In addition, there is no guarantee that any particular version of xunit.runner.visualstudio will use any particular version of xunit.runner.utility. We shipped 2.5.3 of the former at the same time as 2.5.2 of the latter, but even simultaneous shipment isn't guaranteed (and in fact they lived very separate lives back when @clairernovotny owned the VSTest adapter).

When I shipped core framework 2.5.3, it had no runner utility changes that would've necessitated bringing in a new version for the VSTest adapter, so I didn't bother to ship a new one. It appears that when I ship the next core framework drop (which I'm planning to number 2.6.0), this will still be true, so again I'm not likely to ship another VSTest adapter version.

@ViktorHofer
Copy link
Member

ViktorHofer commented Oct 30, 2023 •
edited
Loading

Wow, that's quite a detailed comment. Thank you for spending your time on writing this up. I just reproduced this locally and I don't see our test assemblies or test helper assemblies linking against any of the xunit.runner.* assemblies.

What I see is that because of us wanting to support both xunit.console and VSTest, we bring in xunit.runner.utility.net452.dll from the xunit.console and explicitly exclude the one from xunit.runner.visualstudio which results in that version collision.

This is the only scenario where we end up with potential collisions between versions of the runner utility libraries, so when you're in this scenario, you must very explicitly remain frozen to use the same version of the runner utility libraries as the VSTest adapter is that you're using.

Exactly that.

This begs the question why we ever chose a design that made us copy the xunit.console runner into the test app's output directory. While VSTest requires that, xunit.console shouldn't.

@bradwilson
Copy link

Yeah, that operation makes no sense to me. You were basically just waiting to explode. 😁

Each runner has to bring along their own copy of xunit.runner.utility.* that they're linked against. It's just the VSTest adapter which insists on copying it into your output folder; all the other runners just live elsewhere and are invoked by hand, so the assemblies are never mixed together.

@bradwilson
Copy link

To be even more explicit: xunit.console would never use xunit.runner.utility.* from the test's folder. It would need to live along side xunit.console.exe so that .NET Framework could do the appropriate implicit loading. If you're copying the contents of xunit.console's NuGet package into your test output folder, you should stop doing that. It's unnecessary and counterproductive. Just run it from wherever it lives (in ~/.nuget/packages or some equivalent).

@ViktorHofer
Copy link
Member

you should stop doing that. It's unnecessary and counterproductive. Just run it from wherever it lives (in ~/.nuget/packages or some equivalent).

Just to share some context. We invoke our tests in CI on different machines than the build agents. We use Helix for that, which is a distributed machine testing orchestration infrastructure. Because of that, we can't rely on the package cache. Instead we should have just sent that dependency as an additional "payload" (dependency) to that helix test agent.

Unfortunately our infrastructure here has become quite complex with so many different app models and ways of testing that this change will probably require some time.

But for what is worth, this is indeed our own fault. Thanks for help diagnosing the issue :)

@bradwilson
Copy link

Good luck! 😄

@ViktorHofer ViktorHofer mentioned this issue Jan 8, 2024
Merged
@ericstj
Copy link
Member Author

ericstj commented Jan 8, 2024

@ViktorHofer did you have a suggestion for what needs to happen to fix this?

I gather that by copying xunit runner binaries over to the output directory this is somehow causing the test-discoverer to fail before it gets a chance to examine our test assembly, is that correct?

@ViktorHofer
Copy link
Member

The gist (IIRC) is to not copy xunit.console's assemblies into the output directory and just invoke it from the package cache (as intended). Only copy it into the output directory when archiving the test assemblies for helix submissions (property: /p:ArchiveTests=true).

xunit.runner.visualstudio is designed to be copied into the app output directory but xunit.console isn't. So avoiding the latter makes sense on a local machine.

@bradwilson
Copy link

xunit.runner.visualstudio is designed to be copied into the app output directory

That makes it sound like we do something special here. We don't. The copy is an unfortunate requirement from the VSTest team; if we don't copy the DLLs into the output directory, then whoever runs dotnet test (or equivalent) needs to specify the location of the test adapter path on disk via an additional command line argument.

@bradwilson
Copy link

If it helps, we could add an MSBuild property that we look for to prevent the file copies. They would need to stay on by default but if you know what you're doing, skipping the copies could make sense.

@ViktorHofer
Copy link
Member

I think this really is an issue on our side and needs a fix in our test infrastructure. The xunit.console copying is just wrong and also in-efficient. I should have done this differently in the past.

@ericstj
Copy link
Member Author

ericstj commented Jan 10, 2024 •
edited
Loading

Yeah, other repos don't copy and bundle the runner package as a correlation payload. It's not so hard to make that work, but it does require changes in a few places.

One thing that was odd to me is that we're saying that our binaries are self-consistent and would work if we merely delete files that aren't needed. That tells me we might also have a bug / design-change ask for the dotnet test scenario to not halt when it encounters binding problems in other assemblies in the output directory.

@ViktorHofer
Copy link
Member

It's not so hard to make that work, but it does require changes in a few places.

I was suggesting something simpler for a quick fix. Keep the xunit.console in the app output directory but only when ArchiveTests=true. Otherwise (locally) use the xunit.console from the nuget cache.

@ericstj ericstj mentioned this issue Jan 11, 2024
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jan 11, 2024
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jan 22, 2024
@ericstj ericstj reopened this Jan 24, 2024
@ericstj
Copy link
Member Author

ericstj commented Jan 24, 2024 •
edited
Loading

@akoeplinger found other places that still rely on xunit.console in the output directory.

it looks like #96826 is breaking a couple things

The latter seems more problematic since we document /t:Test in a couple places and I know many people are still in the habit of using it vs dotnet test and in some cases you don't want the vstest runner

It seems we have some more places to root out where we expect the runner in the output directory - those need to either be updated to pick up the runner from the package, or opt-in to copying.

@ViktorHofer
Copy link
Member

The latter seems more problematic since we document /t:Test in a couple places and I know many people are still in the habit of using it vs dotnet test and in some cases you don't want the vstest runner

dotnet build /t:Test should still work with #96826. What are we missing here?

@ViktorHofer ViktorHofer modified the milestones: 9.0.0, 10.0.0 Jul 22, 2024
@ViktorHofer
Copy link
Member

Moving this back into 9.0.0 as this makes debugging .NET Framework tests with dotnet test harder. We can move it back to 10.0.0 if we don't get to it though.

@ericstj ericstj mentioned this issue Aug 20, 2024
Merged
@dotnet-policy-service dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Aug 20, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Sep 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-Infrastructure-libraries dev-innerloop in-pr There is an active PR which will close this issue when it is merged
Projects
None yet
Milestone
9.0.0
3 participants
@bradwilson @ViktorHofer @ericstj