Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot execute tests when DisableAppDomain is true #1598

Closed
avivanoff opened this issue Feb 18, 2023 · 52 comments
Closed

Cannot execute tests when DisableAppDomain is true #1598

avivanoff opened this issue Feb 18, 2023 · 52 comments
Assignees
Milestone

Comments

@avivanoff
Copy link

avivanoff commented Feb 18, 2023

Describe the bug

An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly

Steps To Reproduce

Steps are the same as in #1493.

The problem happens with both 3.0.2 and 4.0.0-preview-20221222-4.
Unfortunately, this time I cannot create a simple project to re-create the problem, but it is reproducible on our pipelines. One thing I noticed Fusion is showing Microsoft.TestPlatform.CoreUtilities is being loaded from test adapter path instead of vstest.console.exe location. Settings DisableAppDomain to false fixes the problem.

The log:

An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources, Version=14.0.0.0, Culture=en-US, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
Stack trace:
Server stack trace: 
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalGetSatelliteAssembly(String name, CultureInfo culture, Version version, Boolean throwOnFileNotFound, StackCrawlMark& stackMark)
   at System.Resources.ManifestBasedResourceGroveler.GetSatelliteAssembly(CultureInfo lookForCulture, StackCrawlMark& stackMark)
   at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(CultureInfo culture, Dictionary`2 localResourceSets, Boolean tryParents, Boolean createIfNotExists, StackCrawlMark& stackMark)
   at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo requestedCulture, Boolean createIfNotExists, Boolean tryParents, StackCrawlMark& stackMark)
   at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo culture, Boolean createIfNotExists, Boolean tryParents)
   at System.Resources.ResourceManager.GetString(String name, CultureInfo culture)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Resource.get_MissingDeploymentDependency()
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.GetDependentAssembliesInternal(String assemblyString, IList`1 result, ISet`1 visitedAssemblies, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.ProcessChildren(Assembly assembly, IList`1 result, ISet`1 visitedAssemblies, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.GetDependentAssembliesInternal(String assemblyString, IList`1 result, ISet`1 visitedAssemblies, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.ProcessChildren(Assembly assembly, IList`1 result, ISet`1 visitedAssemblies, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.GetDependentAssembliesInternal(String assemblyString, IList`1 result, ISet`1 visitedAssemblies, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.ProcessChildren(Assembly assembly, IList`1 result, ISet`1 visitedAssemblies, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.GetFullPathToDependentAssemblies(String assemblyPath, IList`1& warnings)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Deployment.AssemblyLoadWorker.GetFullPathToDependentAssemblies(String assemblyPath, IList`1& warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.AssemblyUtility.GetFullPathToDependentAssemblies(String assemblyPath, String configFile, IList`1& warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.DeploymentUtility.AddDependencies(String testSource, String configFile, IList`1 deploymentItems, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.DeploymentUtility.AddDependenciesOfDeploymentItem(String deploymentItemFile, IList`1 filesToDeploy, IList`1 warnings)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.DeploymentUtilityBase.Deploy(IList`1 deploymentItems, String testSource, String deploymentDirectory, String resultsDirectory)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Utilities.DeploymentUtilityBase.Deploy(String source, IRunContext runContext, ITestExecutionRecorder testExecutionRecorder, IList`1 deploymentItems, TestRunDirectories runDirectories)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestDeployment.Deploy(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.<>c__DisplayClass46_0.<RunTestInternalWithExecutors>b__0()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.<>c__DisplayClass0_0.<Run>b__0()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.Run(Action action, PlatformApartmentState apartmentState, Boolean waitForCompletion)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.TryToRunInStaThread(Action action, Boolean waitForCompletion)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)
Inner exception: Could not load file or assembly 'Microsoft.TestPlatform.CoreUtilities, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.AssemblyResolver.<>c__DisplayClass25_0.<OnResolveInternal>b__0()
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.AssemblyResolver.SafeLog(String assemblyName, Action loggerAction)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.AssemblyResolver.OnResolveInternal(Object senderAppDomain, ResolveEventArgs args, Boolean isReflectionOnly)
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.AssemblyResolver.OnResolve(Object sender, ResolveEventArgs args)
   at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName)

AB#1827543

@avivanoff avivanoff changed the title Cannot eexcute t Cannot execute tests when DisableAppDomain is true Feb 18, 2023
@engyebrahim
Copy link
Member

Hi @avivanoff,
I tried to reproduce the issue I added .runsettings file with app domain setting to true but the tests run without errors (I attached a picture) so please I need a sample project to reproduce the issue.
rr

@avivanoff
Copy link
Author

@engyebrahim, as I said, I could not create a sample project, even though I tried using the exact same settings of our failing one. How can we proceed? Is there a way to produce a detailed log a can send to you?

@Evangelink
Copy link
Member

Hi @avivanoff,

Could you please follow the steps described here https://github.com/microsoft/vstest/blob/main/docs/troubleshooting.md#visual-studio and share the logs with us? It should produce some logs (probably better to clean the folder then run your tests to have the issue).

@avivanoff
Copy link
Author

@Evangelink, I got the logs. I did not remove any possibly sensitive information, so where can I send them?

@Evangelink
Copy link
Member

You can send them to my email amauryleve[at]microsoft[dot]com

@avivanoff
Copy link
Author

@Evangelink, logs are sent. Let me know if this is enough.

@Haplois
Copy link
Contributor

Haplois commented Mar 3, 2023

This might be because of multiple test projects (with different targets / or adapter versions) deploying to same directory.

@avivanoff
Copy link
Author

@Haplois, we use Central Package Management, so all adapters are the same version.

@Haplois
Copy link
Contributor

Haplois commented Mar 4, 2023

@avivanoff - if you provide a small repro, it would be much easier to debug.

Even if you're using the same version of the adapters in all your test projects, if you output all your tests in same directory and they are in different TFNs, that might cause this issue. I'm sure after reviewing the logs @Evangelink will know more.

@avivanoff
Copy link
Author

@Haplois, I did mention in the original description that I was not able to create a sample project with repro this time. I also mentioned that assembly in question is being loaded from the wrong location - test adapter path instead of test host executable path.
All projects target .NET Framework 4.8 AnyCPU.

@Haplois
Copy link
Contributor

Haplois commented Mar 4, 2023

Sorry I read issue around 2am, missed that part. If you send your logs to me I can check the issue, but I'm no longer affiliated with Microsoft. My Twitter DMs are open. (https://twitter.com/medenibaykal)

@MarcoRossignoli
Copy link
Contributor

MarcoRossignoli commented Mar 7, 2023

Unfortunately, this time I cannot create a simple project to re-create the problem, but it is reproducible on our pipelines

@avivanoff are you able to provide your CI setup? How are you running tests(manual vstest.console.exe, VSTest@2 task etc...with some specific runsettings/testadapter path list?)? Are you using Azure DevOps, is so which agent version?

@avivanoff
Copy link
Author

  • Azure DevOps, self-hosted pipeline agents 2.217.2, x64.
  • Command line from the run logs:
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "@C:\BuildAgent\_work\_temp\gaojnzzbvkw.tmp"
2023-03-10T19:21:15.3629877Z Microsoft (R) Test Execution Command Line Tool Version 17.5.0 (x64)
2023-03-10T19:21:15.3631012Z Copyright (c) Microsoft Corporation.  All rights reserved.
2023-03-10T19:21:15.4479151Z vstest.console.exe "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 01>.dll"
2023-03-10T19:21:15.4480338Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 02>.dll"
2023-03-10T19:21:15.4481065Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 03>.dll"
2023-03-10T19:21:15.4481785Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 04>.dll"
2023-03-10T19:21:15.4482441Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 05>.dll"
2023-03-10T19:21:15.4483149Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 06>.dll"
2023-03-10T19:21:15.4483879Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 07>.dll"
2023-03-10T19:21:15.4486428Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 08>.dll"
2023-03-10T19:21:15.4487145Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 09>.dll"
2023-03-10T19:21:15.4487887Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 10>.dll"
2023-03-10T19:21:15.4488611Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 11>.dll"
2023-03-10T19:21:15.4489976Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 12>.dll"
2023-03-10T19:21:15.4490664Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 13>.dll"
2023-03-10T19:21:15.4491421Z "C:\BuildAgent\_work\5\b\x64\Release\<test assembly 14>.dll"
2023-03-10T19:21:15.4492049Z /TestCaseFilter:"TestCategory=CI"
2023-03-10T19:21:15.4492717Z /Settings:"C:\BuildAgent\_work\_temp\c5ogfs2xb5w.tmp.runsettings"
2023-03-10T19:21:15.4493284Z /Logger:"trx"
2023-03-10T19:21:15.4493972Z /TestAdapterPath:"C:\BuildAgent\_work\5\b\x64\Release"
  • Run settings as reported in the run logs:
<RunSettings>
  <RunConfiguration>
    <DisableAppDomain>true</DisableAppDomain> <!--// DEPRECATED: remove when updating to MSTest 4, see https://github.com/microsoft/testfx/issues/1285.-->
    <MaxCpuCount>0</MaxCpuCount>
    <TargetPlatform>x64</TargetPlatform>
    <BatchSize>1000</BatchSize>
    <ResultsDirectory>C:\BuildAgent\_work\_temp\TestResults</ResultsDirectory>
  </RunConfiguration>
  <MSTest>
    <MapInconclusiveToFailed>true</MapInconclusiveToFailed>
    <CaptureTraceOutput>true</CaptureTraceOutput>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>false</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>true</DeploymentEnabled>
    <Parallelize>
      <Workers>0</Workers>
      <Scope>MethodLevel</Scope>
    </Parallelize>
  </MSTest>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="blame" enabled="True">
        <Configuration>
          <ResultsDirectory>C:\BuildAgent\_work\_temp\TestResults</ResultsDirectory>
          <CollectDump />
          <CollectDump />
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
  <LoggerRunSettings>
    <Loggers>
      <Logger friendlyName="blame" enabled="True" />
    </Loggers>
  </LoggerRunSettings>
</RunSettings>

@MarcoRossignoli, any other information I can provide to help you figure out the issue?

@MarcoRossignoli
Copy link
Contributor

MarcoRossignoli commented Mar 13, 2023

@avivanoff is the build machine a dedicated one and it's cleaned at any test run or is it a shared machine with something else? I want to understand if the machine configuration you've specified is complete and we can try to repro it but it means that we don't have to have other software installed on the agent.
Are you using VSTest@2 right?

@avivanoff
Copy link
Author

The problem exists on both VM-based interactive and Docker-based pipeline agents.

Yes, it is VSTest@2 task. But I can also reproduce the issue by manually running the tests.

@avivanoff
Copy link
Author

@Evangelink, @MarcoRossignoli, any updates?

@MarcoRossignoli
Copy link
Contributor

Hi @avivanoff sorry for the late answer, we're working on the repro, no new updates so far.

@avivanoff
Copy link
Author

@MarcoRossignoli, anything? Anything I can do to provide more info?

@avivanoff
Copy link
Author

avivanoff commented Apr 11, 2023

@engyebrahim, @Evangelink, @Haplois, @MarcoRossignoli, as more and more of our pipelines start failing, I was finally able to create a simple sample project TestProject.zip.

  1. Extract to some folder.
  2. Open command line and navigate to that folder.
  3. Build:
    "c:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\amd64\MSBuild.exe" /t:Restore;Build TestProject.sln
  4. Run tests:
    "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "bin\Debug\net48\TestProject1.dll" "bin\Debug\net48\TestProject2.dll" /Settings:"local.runsettings" /TestAdapterPath:"bin\Debug\net48"
  5. Observe the failure.

@MarcoRossignoli
Copy link
Contributor

Can you try to cleanup all bin/obj and bump the version to

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.5.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="3.0.2" />
    <PackageReference Include="MSTest.TestFramework" Version="3.0.2" />
  </ItemGroup>

I've been able to run it in my local

Starting test execution, please wait...
A total of 2 test files matched the specified pattern.
MSTest Executor: Test Parallelization enabled for ...\TestProject\TestProject\TestResults\Deploy_mrossignoli 20230412T085307\Out\TestProject2.dll (Workers: 16, Scope: MethodLevel).
MSTest Executor: Test Parallelization enabled for ...\TestProject\TestProject\TestResults\Deploy_mrossignoli 20230412T085307\Out\TestProject1.dll (Workers: 16, Scope: MethodLevel).
  Passed TestMethod1 [4 ms]
  Passed TestMethod1 [3 ms]

Test Run Successful.
Total tests: 2
     Passed: 2
 Total time: 2.0651 Seconds

@avivanoff
Copy link
Author

That worked. Not sure how I missed these.

But the problem still exists on our build servers. Not sure what to do.

@Evangelink
Copy link
Member

Just to confirm that when running tests locally with the exact same solution/setup the tests are ok but on CI they are failing, right?

If that's the case, that would seem to indicate that there is some difference of setup between the local and the CI.

There is one thing that is puzzling me though. In your first comment you are saying that DisableAppDomain=true causes an issue while DisableAppDomain=false doesn't. The error stack trace is showing calls to System.Runtime.Remoting.Proxies.RealProxy, which would happen through AppDomain marshalling which indicates that the app domains are not disabled.

Would it be possible for us to have access to your real solution setup? I will ask my manager to see if we have some kind of private support path.

@avivanoff
Copy link
Author

@Evangelink,

  • I can reproduce the issue locally.
  • I should be able to send you the content of the bin folder.

@avivanoff
Copy link
Author

@Evangelink, any updates?

@Evangelink
Copy link
Member

Hi @avivanoff,

Sorry for the delay. We weren't able to reproduce the issue in any way.

If you can always reproduce the issue, the easiest would be to have access to the repro (or the source code or dll). You can either send them here, by email to me (amauryleve[at]microsoft[dot]com) or use https://developercommunity.visualstudio.com/home where you can upload privately items.

@avivanoff
Copy link
Author

@Evangelink, I got the email, but there are no attachments or anything else that points to the preview packages.

@Evangelink
Copy link
Member

Evangelink commented May 18, 2023

Maybe our antivirus is doing some cleanup. Let me past link here removed as it was only a test package.

@Evangelink Evangelink added this to the 3.0.3 milestone May 18, 2023
@avivanoff
Copy link
Author

@Evangelink, it worked.

@Evangelink
Copy link
Member

AWESOME! Thank you so much for your patience, I will merge the related PR and we will do the release next week.

@avivanoff
Copy link
Author

@Evangelink, any ETA?

@Evangelink
Copy link
Member

@avivanoff, release is now done. Please let us know if there is any other problem!

@avivanoff
Copy link
Author

@Evangelink, unfortunately, the problem is still there. I am trying to see where my verification went wrong.

Did you try 3.0.3 with the sample binaries I sent you?

@Evangelink
Copy link
Member

@avivanoff, No I haven't tried again as you confirmed the preview package was correct. I am testing now...

@avivanoff
Copy link
Author

@Evangelink, I attached another set of files to reproduce the issue to https://developercommunity.visualstudio.com/t/Cannot-execute-tests-when-DisableAppDoma/10363055.

These are some of my findings:

  • If DisableAppDomain is true and there are deployment items, only the unit test assembly and deployment items are copied to "TestResults\Deploy_XXX\Out". Execution fails.
  • If DisableAppDomain is false and there are deployment items, the unit test assembly, all dependencies and deployment items are copied to "TestResults\Deploy_XXX\Out". Execution succeeds.

In both cases the unit test assembly is loaded from TestResults\Deploy_XXX\Out, while in the first case dependencies are loaded from Bin folder, and in the second case they are loaded from TestResults\Deploy_XXX\Out.

@Evangelink
Copy link
Member

Evangelink commented May 29, 2023

@avivanoff thanks for the extra sample. I have been doing more tests with your binaries and got the same error with all versions of MTest from 2.2.4 to 3.0.2 (I haven't tested previous versions). Could you please let me know if you did some ugprade (if so what was the previously working version?)? Or is it a new test?

PS: I confirm your observations with the 2nd test result (except that tests don't work for me because I don't have the database).

@avivanoff
Copy link
Author

@Evangelink, the tests always worked because they were executing in app domains. It is when I disabled the app domain the tests started to fail.

@Evangelink
Copy link
Member

Thanks for the confirmation! Looks like this bug has always been present (when AppDomains are disabled). I will do another round of tests to be sure I didn't miss anything using your 2 samples and then will proceed with the new release either today or tomorrow

@Evangelink Evangelink modified the milestones: 3.0.3, 3.0.4 May 30, 2023
@avivanoff
Copy link
Author

@Evangelink, any updates on testing and 3.0.4 ETA?

@Evangelink
Copy link
Member

cc @engyebrahim who is handling this release.

@engyebrahim
Copy link
Member

Hello @avivanoff, The release is in progress right now and will let you know after it's done.

@avivanoff
Copy link
Author

avivanoff commented Jun 3, 2023

@Evangelink, @engyebrahim, so far, all test runs that used to fail are succeeding. Thank you.

@Evangelink
Copy link
Member

@avivanoff Glad to hear! Thank you for your patience and for using MSTest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants