NUnit Console 3.15.2: Could not load file or assembly "System.Windows.Forms"

[Kuling]

Hi

I'm in the process of migrating projects to .NET Core 6 from .NET Framework 4.7.2.

I've encountered problem that System.Windows.Forms could not be loaded when tests referencing such library are executed via nunut-console.

However they can be executed without issues via dotnet test or directly from Visual Studio 2022 17.3 Preview 6.

I believe this may be duplicate of #1202

My setup:

• Microsoft.NET.Test.Sdk 17.2.0
• NUnit - 3.13.3
• NUnit3TestAdapter - 4.2.1
• nunit.consolerunner - 3.15.2
• Windows 10 x64

Repository with small project https://github.com/Kuling/nunit-console-test
You can find 2 bat files in root directory which allow to reproduce problem.

After execution of run_via_nunitconsole.bat you shall see

C:\Users\...\source\repos\nunit-console-test>"C:\Users\...\.nuget\packages\NUnit.ConsoleRunner\3.15.2\tools\nunit3-console.exe" "C:\Users\...\source\repos\nunit-console-test\\Test1\bin\Debug\net6.0-windows\Test1.dll"  --framework=netcore-6.0 --labels=Before --noheader --trace=Verbose
Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    C:\Users\...\source\repos\nunit-console-test\\Test1\bin\Debug\net6.0-windows\Test1.dll

=> Test1.Class1.WithFramework
=> Test1.Class1.WithoutFramework

Errors, Failures and Warnings

1) Error : Test1.Class1.WithFramework
System.IO.FileNotFoundException : Could not load file or assembly 'System.Windows.Forms, Version=6.0.2.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. The system cannot find the file specified.
   at Test1.Class1.WithFramework()

Run Settings
    RequestedRuntimeFramework: netcore-6.0
    DisposeRunners: True
    InternalTraceLevel: Verbose
    WorkDirectory: C:\Users\...\source\repos\nunit-console-test
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETCoreApp,Version=v6.0
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    TargetRuntimeFramework: netcore-6.0
    NumberOfTestWorkers: 12

Test Run Summary
  Overall result: Failed
  Test Count: 2, Passed: 1, Failed: 1, Warnings: 0, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 0, Errors: 1, Invalid: 0
  Start time: 2022-08-03 11:56:31Z
    End time: 2022-08-03 11:56:32Z
    Duration: 0.639 seconds

Results (nunit3) saved as TestResult.xml

I suspect that this is related to runtimeconfig (Test1.runtimeconfig.json). https://docs.microsoft.com/en-us/dotnet/core/runtime-config/

{
  "runtimeOptions": {
    "tfm": "net6.0",
    "frameworks": [
      {
        "name": "Microsoft.NETCore.App",
        "version": "6.0.0"
      },
      {
        "name": "Microsoft.WindowsDesktop.App",
        "version": "6.0.0"
      }
    ]
  }
}

I think nunit-console ignores it somehow.
When I manually remove Microsoft.WindowsDesktop.App from above file then dotnet test (executed via run_via_dotnet_test.bat) fails with the same error.

C:\Users\...\source\repos\nunit-console-test>dotnet test "C:\Users\...\source\repos\nunit-console-test\\Test1\bin\Debug\net6.0-windows\Test1.dll"
Microsoft (R) Test Execution Command Line Tool Version 17.3.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved. 
  
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Failed WithFramework [10 ms]
  Error Message:
   System.IO.FileNotFoundException : Could not load file or assembly 'System.Windows.Forms, Version=6.0.2.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. The system cannot find the file specified.
  Stack Trace:
     at Test1.Class1.WithFramework()
  
Failed!  - Failed:     1, Passed:     1, Skipped:     0, Total:     2, Duration: 22 ms - Test1.dll (net6.0)

[Kuling]

I forgot to mention that I also had problems with loading Microsoft.AspNetCore.Hosting.Abstractions for projects with following runtimeconfig.

{
  "runtimeOptions": {
    "tfm": "net6.0",
    "frameworks": [
      {
        "name": "Microsoft.NETCore.App",
        "version": "6.0.0"
      },
      {
        "name": "Microsoft.AspNetCore.App",
        "version": "6.0.0"
      }
    ],
    "configProperties": {
      "System.Reflection.Metadata.MetadataUpdater.IsSupported": false
    }
  }
}`

This is why I think it is duplicate of #1202
However this bug is not a part of repro repository.

[CharliePoole]

@Kuling

The console runner makes no use whatsoever of the runtimeconfig file. Only the runner used by the dotnet command does that.

[Kuling]

If dotnet test command is the only way to run such kind of tests then I'll go that way.
I really like NUnit-console and it was good tool for my needs for many years. I'm reluctant to put it away.

To show my problem in wider scope.

I'm struggling with migration of quite big codebase which uses following toolset:
OpenCover -> NUnitConsole -> NUnit -> ReportGenerator

Alternative for this is as far as I know is dotnet test -> NUnit + coverlet -> ReportGenerator.

Currently I'm not able to run more than 60% of my tests/dlls with nunit-console due to reported issue.
From the other hand approach with dotnet test and coverlet suffers from technical difficulties (lack of parallel test execution, filtering, coverlet scaling).

[CharliePoole]

No, I'm not saying that you can't run your tests - although obviously there's an issue to solve. I'm just taking note of the side issue, which is that you mentioned using runtimesettings, which has only ever been used by the NUnit3 test adapter, not by nunit3-console.exe. The console has always relied on command-line options, which are of course not available to the adapter.

[Kuling]

Possible temporary workaround:

After update of nunit.consolerunner\3.15.2\tools\agents\net6.0\nunit-agent.runtimeconfig.json with following content I was able to resolve problem with loading libraries for mentioned tests.

{
  "runtimeOptions": {
    "tfm": "net6.0",
    "frameworks": [
      {
        "name": "Microsoft.NETCore.App",
        "version": "6.0.0"
      },
      {
        "name": "Microsoft.WindowsDesktop.App",
        "version": "6.0.0"
      },
      {
        "name": "Microsoft.AspNetCore.App",
        "version": "6.0.0"
      }
    ],
    "configProperties": {
      "System.Reflection.Metadata.MetadataUpdater.IsSupported": false
    }
  }
}

[cw397]

We are having the same issue as @Kuling, although with WPF instead of Windows Forms. We are using:

• Microsoft.NET.Test.Sdk 17.1.0
• NUnit 3.13.3
• NUnit.ConsoleRunner 3.15.2

In our case the exception is:
System.IO.FileNotFoundException : Could not load file or assembly 'PresentationCore, Version=6.0.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.

The workaround above by manually editing nunit.consolerunner\3.15.2\tools\agents\net6.0\nunit-agent.runtimeconfig.json to add in Microsoft.WindowsDesktop.App works, but is obviously not ideal.

[CharliePoole]

I'm using @Kuling 's workaround as the base of the solution, modifying the project file so that the nunit-agent.runtimeconfig.json is generated that way. I'll add tests to ensure it keeps working.

[CharliePoole]

I think that the fix will also resolve #1202, but I have kept that open pending confirmation.

[cw397]

The fix (in 3.16.0-dev00043) worked for us for WPF.

[CharliePoole]

@cw397 Thanks for checking!

[CharliePoole]

This issue has been resolved in version 3.16.0

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org

[cw397]

@CharliePoole A slight unintended consequence of this fix... NUnit Console started failing on our test server when we upgraded to 3.16 (previously I'd only tried the fix on a development machine). The reason seems to be that because of the lines that are now present in the nunit-agent.runtimeconfig.json, NUnit Console now always needs the .NET Runtime, .NET Desktop runtime and ASP.NET Runtimes installed. Our test server does not have the ASP.NET runtimes installed so the agent fails to launch.

It could be argued this is another bug? For the time being I have elected to manually edit the nunit-agent.runtimeconfig.json file to remove the ASP.NET Runtime lines, rather than install an additional runtime. Obviously people who use ASP.NET and not .NET Desktop will have the opposite issue, and some people might not need either.

[CharliePoole]

Ouch! I'll take another look at this for 3.16.1.

[CharliePoole]

@cw397 I created issue #1274 to deal with this problem in 3.16.1

[andrewimcclement]

@CharliePoole Is this issue fixed in 3.17.0? I know some weird versioning stuff happened, and I'm seeing what appears to be this issue in 3.17.0.

[CharliePoole]

@andrewimcclement Can you try it in the latest 3.18 dev build? See https://www.myget.org/feed/nunit/package/nuget/NUnit.Console/3.18.0-dev00062

[andrewimcclement]

Confirmed, System.Windows.Forms.dll issue is resolved on the 3.18 dev build.
I can't claim everything is working (the runner crashes due to a SocketException):

System.Net.Sockets.SocketException : An existing connection was forcibly closed by the remote host

--SocketException
An existing connection was forcibly closed by the remote host
   at NUnit.Engine.Communication.Transports.Tcp.TestAgentTcpProxy.SendCommandMessage(String command, Object[] arguments)
   at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing)
   at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
   at NUnit.Engine.Runners.MasterTestRunner.Dispose(Boolean disposing)
   at NUnit.Engine.Runners.MasterTestRunner.Dispose()
   at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
   at NUnit.ConsoleRunner.Program.Main(String[] args)

However, until that point (which could be caused by my own code) I can see several tests running successfully (with both successes and failures). So I think this specific issue would count as resolved in the 3.18 dev build (I might just need to raise more issues, potentially).

[andrewimcclement]

For now I am just switching over to using dotnet test (losing the speed of nunit3-console.exe is annoying, but correctness has to come first, of course).