On TestCases...

[JustinGrote]

I noticed that testcases return multiple results in both NoRun and Run mode.

Two questions:

1. Why can't the label of the test be updated to interpolate wildcards to the actual name of the value used?
2. Can the exception TargetObject be updated to point to the line where the testcase originated? This would make it easier to lens to which particular test case is not working.

Example:

Describe "Various Test Styles" {
    # TODO: Handle this scenario. Today it currently just accepts the first result
    It "Multiple TestCases Array <Name>" {
        $Name | Should -BeOfType [int]
    } -TestCases @(
        @{Name = 1}
        @{Name = 'Should Fail'}
        @{Name = 2}
    )
}

Translated JSON Output by PesterInterface.ps1

[
  {
    "type": "test",
    "id": "C:\\Users\\JGrote\\Projects\\vscode-pester-test-adapter\\sample\\Tests\\TestStyles.Tests.ps1:3",
    "file": "C:\\Users\\JGrote\\Projects\\vscode-pester-test-adapter\\sample\\Tests\\TestStyles.Tests.ps1",
    "startLine": 2,
    "endLine": 4,
    "label": "Multiple TestCases Array <Name>",
    "result": 3,
    "duration": 8,
    "message": null,
    "expected": null,
    "actual": null,
    "targetFile": null,
    "targetLine": -1
  },
  {
    "type": "test",
    "id": "C:\\Users\\JGrote\\Projects\\vscode-pester-test-adapter\\sample\\Tests\\TestStyles.Tests.ps1:3",
    "file": "C:\\Users\\JGrote\\Projects\\vscode-pester-test-adapter\\sample\\Tests\\TestStyles.Tests.ps1",
    "startLine": 2,
    "endLine": 4,
    "label": "Multiple TestCases Array <Name>",
    "result": 4,
    "duration": 47,
    "message": "Expected the value to have type [int] or any of its subtypes, but got 'Should Fail' with type [string].",
    "expected": "the value to have type [int] or any of its subtypes",
    "actual": "'Should Fail' with type [string]",
    "targetFile": "C:\\Users\\JGrote\\Projects\\vscode-pester-test-adapter\\sample\\Tests\\TestStyles.Tests.ps1",
    "targetLine": 3
  },
  {
    "type": "test",
    "id": "C:\\Users\\JGrote\\Projects\\vscode-pester-test-adapter\\sample\\Tests\\TestStyles.Tests.ps1:3",
    "file": "C:\\Users\\JGrote\\Projects\\vscode-pester-test-adapter\\sample\\Tests\\TestStyles.Tests.ps1",
    "startLine": 2,
    "endLine": 4,
    "label": "Multiple TestCases Array <Name>",
    "result": 3,
    "duration": 2,
    "message": null,
    "expected": null,
    "actual": null,
    "targetFile": null,
    "targetLine": -1
  }
]

[fflaten]

1. Why can't the label of the test be updated to interpolate wildcards to the actual name of the value used?

The lazy variables may reference other variables available at runtime than those in testcase, so we'd have to run all BeforeAll/BeforeEach setups as well which can be time-consuming. SkipRun currently skips the entire Run-phase to avoid that.

2. Can the exception TargetObject be updated to point to the line where the testcase originated?

Testcases can be an array that's reused in multiple tests, even with similar or the same name. Wouldn't that make it unclear which test you're referencing?

I do see you problem though as discussed earlier. It's hard to reference data-driven tests based due to script parameters and x levels of foreach/testcases. Best option is likely containerId:startline:testcaseindex (containerId being scritblock-id or path) and hope it's not so dynamic it changes on the next run. 😁

[JustinGrote]

For my purposes that's fine, it's very loose tracking based on a unique ID. However, based on the pester result object, for those individual test case objects, I have no idea how to distinguish between them. What object in the Pester class would work?

[JustinGrote]

It turns out that when doing discovery with NoRun, the Data property is populated! So it is possible to distinguish testcases. I made a function to generate unique test IDs even for TestCase Items. Lots of edge case issues but its good enough for now.

Function

function New-TestItemId {
    <#
    .SYNOPSIS
    Create a string that uniquely identifies a test or test suite
    .NOTES
    Can be replaced with expandedpath if https://github.com/pester/Pester/issues/2005 is fixed
    #>
    [CmdletBinding()]
    param(
        [Parameter(Mandatory,ValueFromPipeline)]
        [ValidateScript({
            $null -ne ($_.PSTypeNames -match '^(Pester\.)?Test$')
        })]$Test
    )
    process {
        if ($Test.Path -match '\|') {
            throw [NotSupportedException]"The pipe character '|' is not supported in test names with this adapter. Please remove all pipes from test/context/describe names"
        }

        #TODO: Edge cases
        #Non-String Object
        #Non-STring Hashtable
        #Other dictionaries
        #Nested Hashtables
        #Fancy TestCases, maybe just iterate them as TestCaseN or exclude
        $Data = $Test.Data
        if ($Data) {
            if ($Data -is [IDictionary]) {
                $Test.Data.GetEnumerator() | Sort-Object Name | Foreach-Object {
                    $Test.Path += [String]([String]$PSItem.Name + '=' + [String]$PSItem.Value)
                }
            } else {
                $Test.Path += [string]$Data
            }
        }

        $Test.Path -join '|'
    }

}

Result on several tests including testcases.

➜ (invoke-pester -Configuration @{Run=@{SkipRun=$true;PassThru=$true}}).tests | New-TestItemId

Test|test
Basic|Describe-Level Succeeds
Basic|Describe-Level Fails
Basic|Succeeds|True
Basic|Succeeds|False
Basic|Succeeds|ShouldBeTrue
Basic|Fails|Throws
Basic|Fails|ShouldThrow
Basic|Fails|ShouldBeTrue
Basic|Fails|ShouldBeGreaterThan
TestCases|TestCase Array <_>|1
TestCases|TestCase Array <_>|2
TestCases|TestCase Array <_>|red
TestCases|TestCase Array <_>|blue
TestCases|TestCase HashTable <Name>|Name=1
TestCases|TestCase HashTable <Name>|Name=2
TestCases|TestCase HashTable <Name>|Name=red
TestCases|TestCase HashTable <Name>|Name=blue
Duplicate DescribeWithContext|DupeContext|DupeContext
Duplicate DescribeWithContext|DupeContext|DupeContext

[nohwnd]

And you should also be able to order tests based on their position in test cases. Even though that looks like it is working already pretty well.

[JustinGrote]

@nohwnd yeah part of my goal is that test history is tracked, and I wanted to preserve that as much as possible if a test "moves" which is why I wanted to go beyond simple file:line matching.

[JustinGrote]

@fflaten haven't tried it yet, right now I'm just dropping duplicates with an error message.

[JustinGrote]

@fflaten it doesn't populate the data but I noticed .block.data has the right info so I think I can make it work, pending...

[JustinGrote]

@fflaten got it to work, the unicode characters aren't coming over but that's a separate issue.

[JustinGrote]

Implemented in JustinGrote/_PR-vscode-pester-test-adapter@3e6f6b9

And I am able to correlate the tests together using my constructed id.