Testhost stays alive even though KeepAlive=false

[riQQ]

Description

Testhost stays alive indefinitely on a CannotUnloadAppDomainException even though KeepAlive is false and therefore vstest.console.exe doesn't exit.

9328, 11, 2023/11/23, 10:40:46.220, 3043828188124, testhost.net472.x86.exe, Cannot unload app-domain. Reason: System.CannotUnloadAppDomainException: Error while unloading appdomain. (Exception from HRESULT: 0x80131015)
   at System.AppDomain.Unload(AppDomain domain)
   at Microsoft.VisualStudio.TestPlatform.MSTestFramework.UnitTestAdapter.Dispose(Boolean isExplicitDispose)
   at Microsoft.VisualStudio.TestPlatform.MSTestFramework.UnitTestAdapter.Dispose()
   at Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration.MixedModeExecutor.ExecuteDurontoTestsInternal(IEnumerable`1 tests, IRunContext runContext, ITestExecutionRecorder testExecutionRecorder, String source, TestRunDirectories runDirectories, TmiTestRun tmiTestRun, Int32 globalTestTimeout)
   at Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration.MixedModeExecutor.ExecuteDurontoTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, String source, TestRunDirectories runDirectories, TmiTestRun tmiTestRun, Int32 globalTestTimeout)
9328, 11, 2023/11/23, 10:40:46.222, 3043828193686, testhost.net472.x86.exe, Not hinting the framework to shutdown the process after the run is complete

Related issue: microsoft/testfx#225

Steps to reproduce

Have tests that causes a CannotUnloadAppDomainException on disposing the UnitTestAdapter and the Testhost just stays alive indefinitely and therefore vstest.console.exe doesn't exit.

Expected behavior

Testhost exits with error on encountering the exception.

Actual behavior

Testhost stays alive indefinitely.

Diagnostic logs

        "TestExecutionContext": {
            "FrequencyOfRunStatsChangeEvent": 10,
            "RunStatsChangeEventTimeout": "00:00:01.5000000",
            "InIsolation": false,
            "KeepAlive": false,
            "AreTestCaseLevelEventsRequired": false,
            "IsDebug": false,
            "TestCaseFilter": "<redacted-filters>",
            "FilterOptions": null
        },

TpTrace Verbose: 0 : 9328, 1, 2023/11/23, 10:13:21.554, 3038540892462, testhost.net472.x86.exe, Version: 17.4.1-release-20221129-02 Current process architecture: X86
TpTrace Verbose: 0 : 9328, 1, 2023/11/23, 10:13:21.563, 3038540920179, testhost.net472.x86.exe, Runtime location: C:\Windows\Microsoft.NET\Framework\v4.0.30319
TpTrace Information: 0 : 9328, 1, 2023/11/23, 10:13:21.573, 3038540952390, testhost.net472.x86.exe, DefaultEngineInvoker.Invoke: Testhost process started with args :[--port, 53096],[--endpoint, 127.0.0.1:053096],[--role, client],[--parentprocessid, 6604],[--diag, <redacted-dir>\VSTest\vstest_diag.host.23-11-23_10-13-21_30742_6.log],[--tracelevel, 4],[--datacollectionport, 53095],[--telemetryoptedin, false]
...
TpTrace Information: 0 : 9328, 8, 2023/11/23, 10:13:21.934, 3038542113688, testhost.net472.x86.exe, TestRequestHandler.OnMessageReceived: received message: (TestExecution.StartWithSources) -> 
...
9328, 8, 2023/11/23, 10:40:45.888, 3043827116369, testhost.net472.x86.exe, TcpClientExtensions.MessageLoopAsync: Polling on remoteEndPoint: 127.0.0.1:53096 localEndPoint: 127.0.0.1:53097
9328, 11, 2023/11/23, 10:40:46.220, 3043828188124, testhost.net472.x86.exe, Cannot unload app-domain. Reason: System.CannotUnloadAppDomainException: Error while unloading appdomain. (Exception from HRESULT: 0x80131015)
   at System.AppDomain.Unload(AppDomain domain)
   at Microsoft.VisualStudio.TestPlatform.MSTestFramework.UnitTestAdapter.Dispose(Boolean isExplicitDispose)
   at Microsoft.VisualStudio.TestPlatform.MSTestFramework.UnitTestAdapter.Dispose()
   at Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration.MixedModeExecutor.ExecuteDurontoTestsInternal(IEnumerable`1 tests, IRunContext runContext, ITestExecutionRecorder testExecutionRecorder, String source, TestRunDirectories runDirectories, TmiTestRun tmiTestRun, Int32 globalTestTimeout)
   at Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration.MixedModeExecutor.ExecuteDurontoTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, String source, TestRunDirectories runDirectories, TmiTestRun tmiTestRun, Int32 globalTestTimeout)
9328, 11, 2023/11/23, 10:40:46.222, 3043828193686, testhost.net472.x86.exe, Not hinting the framework to shutdown the process after the run is complete
9328, 11, 2023/11/23, 10:40:46.223, 3043828197950, testhost.net472.x86.exe, Deleting deployment directory <redacted-path>\TestResults\<redacted-deploy-dir>
9328, 11, 2023/11/23, 10:40:46.583, 3043829349487, testhost.net472.x86.exe, DeploymentManager.DeleteDirectories failed for the directory '<redacted-path>\TestResults\<redacted-deploy-dir>': System.UnauthorizedAccessException: Access to the path 'Microsoft.Expression.Interactions.dll' is denied.
   at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound, WIN32_FIND_DATA& data)
   at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
   at System.IO.DirectoryInfo.Delete(Boolean recursive)
   at Microsoft.VisualStudio.TestPlatform.MSTestFramework.DeploymentManager.DeleteDirectories(String filePath)
9328, 11, 2023/11/23, 10:40:46.584, 3043829353227, testhost.net472.x86.exe, Deleted deployment directory <redacted-path>\TestResults\<redacted-deploy-dir>
9328, 8, 2023/11/23, 10:40:46.890, 3043830336582, testhost.net472.x86.exe, TcpClientExtensions.MessageLoopAsync: Polling on remoteEndPoint: 127.0.0.1:53096 localEndPoint: 127.0.0.1:53097

Environment

Windows Server x64
vstest.console.exe, Version: 17.4.1-release-20221129-02 Current process architecture: X64

[nohwnd]

This is a common problem and unfortunately there is nothing we can do about that error. You can try disabling appdomains unless you depend on them. This will get rid of the error and might even make your test run much faster.

Here is how: #4726

[riQQ]

I know that the error is our mistake and caused by our code. Thanks for the hint about disabling appdomains. But that is merely a workaround.

This ticket is about the KeepAlive option, my understanding is that the Testhost should exit on encountering the exception because of KeepAlive = false. Is this notion correct? I couldn't find any documentation about the exact meaning of the KeepAlive option.

I only found that KeepAlive is always set to false when running vstest.console.exe:

vstest/src/vstest.console/Processors/RunTestsArgumentProcessor.cs

Lines 168 to 172 in a86e108

	// for command line keep alive is always false.
	// for Windows Store apps it should be false, as Windows Store apps executor should terminate after finishing the test execution.
	var keepAlive = false;
	var runRequestPayload = new TestRunRequestPayload() { Sources = _commandLineOptions.Sources.ToList(), RunSettings = runSettings, KeepAlive = keepAlive, TestPlatformOptions = new TestPlatformOptions() { TestCaseFilter = _commandLineOptions.TestCaseFilterValue } };

[nohwnd]

KeepAlive is for shared testhosts, which is a concept in .NET Framework run where you can isolate the tested dlls from each other using appdomains, so it is also possible to run multiple test runs, using multiple different dlls in the same process, and keep alive is avoiding the overhead of closing and starting a new process when you technically don't need to.

https://grep.app/search?q=KeepAlive&filter[repo][0]=microsoft/vstest

It should not affect whether or not the testhost stays alive in case of error.

When there is appdomain unload failure the process cannot finish, and there is little that we can do, at least not without changing the flow, to kill the process.

[riQQ]

Okay, so I misunderstood what KeepAlive means. Thanks for clearing that up.

When there is appdomain unload failure the process cannot finish, and there is little that we can do, at least not without changing the flow, to kill the process.

Could you expand a little bit on this? Why can't this be changed?

[nohwnd]

The current flow does not allow to work around appdomain unloading problems in a reliable way, if you can disable running with appdomains using one of those ways:

#4726 (comment)