Move tests to invoke te.exe directly instead of using VSTest runner

[miniksa]

Moves the tests from using the vstest.console.exe route to just using te.exe.

PROs:

• te.exe is significantly faster for running tests because the TAEF/VSTest adapter isn't great.
• Running through te.exe is closer to what our developers are doing on their dev boxes
• te.exe is how they run in the Windows gates.
• te.exe doesn't seem to have the sporadic 0x6 error code thrown during the tests where somehow the console handles get lost
• te.exe doesn't seem to repro the other intermittent issues that we have been having that are inscrutable.
• Fewer processes in the tree (te is running anyway under vstest.console.exe, just indirected a lot
• The log outputs scroll live with all our logging messages instead of suppressing everything until there's a failure
• The log output is actually in the order things are happening versus vstest.

CONs:

• No more code coverage.
• No more test records in the ADO build/test panel.
• Tests really won't work inside Visual Studio at all.
• The log files are really big now
• Testing is not a test task anymore, just another script.

Refuting each CON:

• We didn't read the code coverage numbers
• We didn't look at the ADO test panel results or build-over-build velocities
• Tests didn't really work inside Visual Studio anyway unless you did the right incantations under the full moon.
• We could tone down the logging if we wanted at either the te.exe execution time (with a switch) or by declaring properties in the tests/classes/modules that are very verbose to not log unless it fails.
• I don't think anyone cares how they get run as long as they do.

[DHowett-MSFT]

/me reviews it

[miniksa]

K. I know this is bad. I need to use a var to determine the config/platform and/or a PowerShell script that can replace the pattern ** search from what the VS test task could do.

[miniksa]

vstest.console.exe runtime x86 unit tests: 5m8s
te.exe runtime x86 unit tests: 46s

Yeah, uh, let's just run them direct.

[DHowett-MSFT]

# EARLIER

- task: PowerShell@2
  displayName: 'Rationalize build platform'
  inputs:
    targetType: inline
    script: |
      $Arch = "$(BuildPlatform)"
      If ($Arch -Eq "x86") { $Arch = "Win32" }
      Write-Host "##vso[task.setvariable variable=RationalizedBuildPlatform]${Arch}"

# LATER

- task: PowerShell@2
  displayName: 'Run Unit Tests'
  inputs:
    targetType: inline
    script: |
      $TestRoot = ".\bin\$(RationalizedBuildPlatform)\$(BuildConfiguration)"
      & ${TestRoot}\TE.exe ${TestRoot}\*.Unit.Test*.dll
  condition: bla bla bla

perhaps? Maybe?

[miniksa]

feature tests vstest.console.exe: 2m42s
feature tests te.exe: 1m39s

[miniksa]

# EARLIER

- task: PowerShell@2
  displayName: 'Rationalize build platform'
  inputs:
    targetType: inline
    script: |
      $Arch = "$(BuildPlatform)"
      If ($Arch -Eq "x86") { $Arch = "Win32" }
      Write-Host "##vso[task.setvariable variable=RationalizedBuildPlatform]${Arch}"

# LATER

- task: PowerShell@2
  displayName: 'Run Unit Tests'
  inputs:
    targetType: inline
    script: |
      $TestRoot = ".\bin\$(RationalizedBuildPlatform)\$(BuildConfiguration)"
      & ${TestRoot}\TE.exe ${TestRoot}\*.Unit.Test*.dll
  condition: bla bla bla

perhaps? Maybe?

Yeah that looks a bit better than what I came up with.

[oising]

ERMAGAHD

[DHowett-MSFT]

nit: change name to be imperative (Move, Do, Fix, Teach, etc.)

[ghost]

Hello @DHowett-MSFT!

Because this pull request has the AutoMerge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

Do note that I've been instructed to only help merge pull requests of this repository that have been opened for at least 8 hours, a condition that will be fulfilled in about 4 hours 33 minutes. No worries though, I will be back when the time is right! 😉

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

[zadjii-msft]

heres the build

I'm inclined to sign off anyways for the speed boost, but it looks like the roundtrip tests still hang in just TAEF on CI 😢

[DHowett-MSFT]

At least we can now see which one was running when it hangs! When DevOps actually lets you scroll continuously (instead of jumping from line 110000 to line 100025...)

[DHowett-MSFT]

2020-02-06T23:53:14.6908403Z StartGroup: ConptyOutputTests::WriteAFewSimpleLines
2020-02-06T23:53:14.6911125Z Write more lines of output. We should use 
2020-02-06T23:53:14.6911809Z  to move the cursor
2020-02-06T23:53:14.6912996Z Verify: IsNotNull(_pVtRenderEngine.get())
2020-02-06T23:53:14.6914899Z Verify: IsGreaterThan(expectedOutput.size(), static_cast<size_t>(0)): writing="�[2J", expecting 4 strings
2020-02-06T23:53:14.6915910Z Expected =	"�[2J"
2020-02-06T23:53:14.6917308Z Actual =	"�[2J"

[miniksa]

Yes, I will go dig into that test now and fixing it or turning it off (with a TODO) may be a part of this too.

[miniksa]

This build is running 1767 unit tests and 405 feature tests (another PR): https://dev.azure.com/ms/Terminal/_build/results?buildId=62323&view=logs&j=0f660c0a-4423-5c5d-276b-f98b39c69e13&t=ecf341a7-dafd-5ce6-eabd-86286d881b78

This build that just ran has 1753 unit tests and 405 feature test: https://dev.azure.com/ms/Terminal/_build/results?buildId=62403&view=logs&j=0f660c0a-4423-5c5d-276b-f98b39c69e13&t=f0309e56-3cac-5248-a471-a2958657f90a

I think I'm missing one module...

[miniksa]

Yeah I'm missing TerminalAppUnitTests because they're in a bin subdirectory...

[miniksa]

OK it blended mostly. Take out the failboat test and update the solution. If this works, we're good to go.

[DHowett-MSFT]

Get them good-good signoffs!