dotnet test on linux fails to find tests

[mjkkirschner]

Description

Unclear if this is an Nunit or dotnet test issue, but starting here as Nunit seems to report the correct number of tests.

When running under linux - ubuntu 22.04 locally or on github hosted runner - the command:
dotnet test pathTotestBinary.dll fails to run any tests. This binary references nunit3. The nunit framework, engine, and test adapter binaries are in the same directory as the test binary.

Interestingly, in verbose mode, nunit test adapter / executer reports that it finds all the tests.

Also -t list mode shows all the tests.

This same command works fine on windows.

dotnet 6.0.406 is installed locally.

output looks like:

parallels@parallels-Parallels-Virtual-Platform:~/Documents/Dynamo$ dotnet test bin/NET60_Linux/Release/DynamoCoreTests.dll -v:d --diag:dotnettestlog.txt
Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Logging Vstest Diagnostics in file: /home/parallels/Documents/Dynamo/dotnettestlog.txt
A total of 1 test files matched the specified pattern.
/home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll
Logging TestHost Diagnostics in file: /home/parallels/Documents/Dynamo/dotnettestlog.host.23-02-27_19-52-44_18198_5.txt
NUnit Adapter 4.2.0.0: Test execution started
Running all tests in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll
   NUnit3TestExecutor discovered 2568 of 2568 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.2.0.0: Test execution complete
No test is available in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

Steps to reproduce

dotnet 6 / 7
nunit 3 adapter 4.2.1
nunit 13.3

Expected behavior

~2500 tests should be found

Actual behavior

dotnet test reports there are not tests in the assembly.

Diagnostic logs

dotnettestlog.zip

Environment

using parallels locally, but issue reproduces on github ubuntu runner.

[mjkkirschner]

I've noticed that this same message is sometimes shown when test dependencies are missing (like Moq) is that expected?

[Evangelink]

Hi @mjkkirschner,

It would be good to have access to the project configuration to be able to tell for sure but from the logs it would seem to be an issue with NUnit.

TpTrace Information: 0 : 27309, 4, 2023/02/27, 19:52:34.684, 4441635620303, testhost.dll, Running all tests in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:34.684, 4441635972496, testhost.dll, TestRequestHandler.SendData: sending data from testhost: {"Version":7,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"Running all tests in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll"}}
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:34.686, 4441637991471, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestRunMessageLoggerProxy.SendMessage: Invoking callback 1/1 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.OnTestRunMessage, took 2 ms.
TpTrace Information: 0 : 27309, 4, 2023/02/27, 19:52:35.162, 4442113309016, testhost.dll,    NUnit3TestExecutor discovered 2568 of 2568 NUnit test cases using Current Discovery mode, Non-Explicit run
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.162, 4442113752886, testhost.dll, TestRequestHandler.SendData: sending data from testhost: {"Version":7,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"   NUnit3TestExecutor discovered 2568 of 2568 NUnit test cases using Current Discovery mode, Non-Explicit run"}}
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.164, 4442115584495, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestRunMessageLoggerProxy.SendMessage: Invoking callback 1/1 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.OnTestRunMessage, took 2 ms.
TpTrace Information: 0 : 27309, 4, 2023/02/27, 19:52:35.218, 4442170229525, testhost.dll, NUnit Adapter 4.2.0.0: Test execution complete
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.219, 4442170736460, testhost.dll, TestRequestHandler.SendData: sending data from testhost: {"Version":7,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"NUnit Adapter 4.2.0.0: Test execution complete"}}
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.220, 4442171334186, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestRunMessageLoggerProxy.SendMessage: Invoking callback 1/1 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.OnTestRunMessage, took 1 ms.
TpTrace Verbose: 0 : 27309, 4, 2023/02/27, 19:52:35.231, 4442182347272, testhost.dll, BaseRunTests.RunTestInternalWithExecutors: Completed running tests for executor://NUnit3TestExecutor
TpTrace Warning: 0 : 27309, 4, 2023/02/27, 19:52:35.232, 4442183454544, testhost.dll, No test is available in /home/parallels/Documents/Dynamo/bin/NET60_Linux/Release/DynamoCoreTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

As you can see, the testhost is starting the test execution, then NUnit says it found some tests an then says it's done invoking them but the test platform didn't receive any test result hence the message.

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 10 days.

[mjkkirschner]

Hi @Evangelink

thanks for the reply, to reproduce the issue you can checkout this github action:
https://github.com/DynamoDS/Dynamo/blob/master/.github/workflows/dynamoNet6.0_linux_build.yml
note that the tests are currently disabled due to the above issue.

the test assembly csproj:
https://github.com/DynamoDS/Dynamo/blob/master/test/DynamoCoreTests/DynamoCoreTests.csproj

Please let me know if that was not what you meant by access to the project configuration.

[nohwnd]

This was hopefully solved in the meantime, we don't have others reporting this problem. If not let me know.

[varajan]

Reproduced on my project. @nohwnd, could you please reopen the issue?

dotnet test <project.csproj> --filter TestCategory=Smoke

windows-2022: tests are found;
ubuntu: No test is available in Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

<PackageVersion Include="NUnit" Version="4.1.0" />
<PackageVersion Include="NUnit.Analyzers" Version="4.2.0" />
<PackageVersion Include="NUnit3TestAdapter" Version="4.5.0" />
<PackageVersion Include="NUnit.ConsoleRunner" Version="3.18.1" />

[nohwnd]

@OsirisTerje do you know if this is a known issue in NUnit? Before I spend time investigating.

[nohwnd]

I don't see it reproduce locally in WSL, would you able to provide logs from --diag:log.txt ?

[OsirisTerje]

1. Is this about the same issue as No tests found in large test assembly on linux, ~2500 found on windows. nunit/nunit#4439 ? In that case it was loading issues, not related to NUnit. And was closed as "not a NUnit issue".
2. One should not need both the ConsoleRunner and the Adapter. Also, they may have different versions of the engine, which is embedded in both. See https://docs.nunit.org/articles/vs-test-adapter/Adapter-Engine-Compatibility.html . In this case the adapter should be updated to 4.6, OR, the console downgraded to 3.15.4. From what I can see of the build, it looks like the ConsoleRunner is not needed. But it seems they also use TestComplete, so not sure how all this is connected together.
3. If tests are found using Windows, and not Linux, I don't think this is NUnit related. Only thing would be what I mention in (2)

@varajan This is the same project, right? If not, do you have a repro project?

[varajan]

@varajan This is the same project, right? If not, do you have a repro project?

It's my working project. Unfortunately, I cannot share it. It's running on AzureDevOps, filtering returned 'no tests found' when we've used 'ubuntu' machines and worked correctly when switched to 'windows'. Also, it worked on 'ubuntu' when changed version of nUnit from 4.1 to 3.14

[OsirisTerje]

@varajan Suggest you open a new issue in the NUnit repo, and create a small repro which you can upload to the NUnit.Issues repo. That way we can check if this is something related to NUnit.