Catastrophic failure: System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'X:\app\publish\MyIntegrationTests.exe' with working directory 'X:\app\publish'. The system cannot find the file specified.

[gieniowski]

I found this issue when running integration tests in docker container (mcr.microsoft.com/dotnet/sdk:8.0-alpine) but it seems like it does not matter.

Let's start with publishing binaries with this command:

dotnet publish "MyIntegrationTests.csproj" -c Release -o /app/publish /p:UseAppHost=false

Note the UseAppHost=false (this will not produce executable file, only MyIntegrationTests.dll).

Next I want to run the tests. Here is the command with result:

X:\app\publish>dotnet test MyIntegrationTests.dll --logger "trx;LogFileName=testresults.trx"
VSTest version 17.12.0-preview-24412-03 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.06] MyIntegrationTests: Catastrophic failure: System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'X:\app\publish\MyIntegrationTests.exe' with working directory 'X:\app\publish'. The system cannot find the file specified.
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at Xunit.v3.LocalTestProcess.Start(String executable, String executableArguments, String responseFile)
   at Xunit.v3.LocalOutOfProcessTestProcessLauncher.StartTestProcess(String executable, String executableArguments, String responseFile)
   at Xunit.v3.OutOfProcessTestProcessLauncherBase.Launch(XunitProjectAssembly projectAssembly, IReadOnlyList`1 arguments)
   at Xunit.Runner.v3.Xunit3..ctor(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.v3.Xunit3.ForDiscoveryAndExecution(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.XunitFrontController.Create(XunitProjectAssembly projectAssembly, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink, ITestProcessLauncher testProcessLauncher)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSink reporterMessageHandler, AssemblyRunInfo runInfo)
   at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSink reporterMessageHandler, AssemblyRunInfo runInfo)
No test is available in X:\app\publish\MyIntegrationTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Results File: X:\app\publish\TestResults\testresults.trx

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

Apparently runner did not found MyIntegrationTests.exe, but I specified that I want to execute tests from MyIntegrationTests.dll (exe is not created). I can workaround it by creating exe file (UseAppHost=true) but it is not recommended approach with containers.

Below you can find nuget versions that I use:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <WarningsAsErrors>nullable</WarningsAsErrors>
    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="3.0.0-pre.35">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="xunit.v3" Version="0.4.0-pre.20" />
  </ItemGroup>

</Project>

[bradwilson]

Note the UseAppHost=false (this will not produce executable file, only MyIntegrationTests.dll).

Oh, just saw this. This is definitely not supported. We require the app host to be present.

[gieniowski]

This is all fine if it is intended and it seems like it is breaking change in upgrade from xunit 2.8.2 to v3.

I am not sure if it should throw error though... Messaging is also inconsistent, as I ask for dll file dotnet test MyIntegrationTests.dll and error says An error occurred trying to start process 'X:\app\publish\MyIntegrationTests.exe' and then the message No test is available in X:\app\publish\MyIntegrationTests.dll is clearly wrong as I have tests in this dll file.

Just to be clear, dotnet test command is executed in directory with published artifacts, csproj file is not available here.

[bradwilson]

This is all fine if it is intended and it seems like it is breaking change in upgrade from xunit 2.8.2 to v3.

It's definitely a change from v2 to v3. In v2, test projects are class libraries; in v3, they're programs. The app host is necessary for us to launch the program.

I am not sure if it should throw error though...

It's gonna throw for sure. The only question is, should I catch and throw something more meaningful that might say something like "I couldn't find the executable, make sure you didn't disable the app host"?

[bradwilson]

the message No test is available in X:\app\publish\MyIntegrationTests.dll is clearly wrong

That I can't do anything about. That's VSTest saying "you tried to run something and nothing ran". That will continue to be the case even if the exception that gets thrown has a different message.

[bradwilson]

Message updated in v3 0.4.0-pre.21 (for xunit.v3.runner.console) and xunit.runner.visualstudio 3.0.0-pre.36 (for dotnet test) https://xunit.net/docs/using-ci-builds

This is the failure you will see now with dotnet test:

[xUnit.net 00:00:00.04] empty: Catastrophic failure: System.InvalidOperationException: Could not find app host executable '/path/to/testAssembly.exe'. Make sure you did not disable the app host when building the test project.

[gieniowski]

Thank you for explaining how things work now. New message should also help a lot!