Pester throws an exception when run inside runspace

[kborowinski]

Checklist

• Issue has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

(This issue was moved from here)

Environment:

1. PowerShell 7.3.6 on up-to-date Windows 10 22H2 x64
2. Pester 5.5.0
3. ThreadJob 2.0.3 and Microsoft.PowerShell.ThreadJob 2.1.0

Explanation:
I use Pester for advanced infrastructure testing. Due to the long execution time of some tests I need to run them in parallel using Start-ThreadJob cmdlet from ThreadJob module. The tests are independent of each other. Sporadically, during initialization, Pester throws an exception:
Cannot index into a null array.

The line 13998 in Pester.psm1 module where exceptions happens:

Expected Behavior

Pester should not throw when running in an runspace.

Steps To Reproduce

1. Start new session:
   pwsh -nop

2. Run following scriptblock:

$cpuCount = [Environment]::ProcessorCount

$splatStartThreadJob = @{
    ThrottleLimit = $cpuCount
    StreamingHost = $Host
    ErrorAction = 'Stop'
    InitializationScript = {Import-Module Pester -Force -ErrorAction Stop}
}

$threadJobs = 1..($cpuCount * 2) | ForEach-Object {
    Start-ThreadJob -Name $_ -ScriptBlock {
        $pc = New-PesterConfiguration
        $pc.Run.Container = New-PesterContainer -ScriptBlock {Describe 'd' { It 'i' { 1 | Should -Be 1 }}}
        $pc.Output.Verbosity = 'None'
        $pc.Run.PassThru = $true
        $pc.Run.SkipRemainingOnFailure ='Block'
        [PSCustomObject]@{
            Test    = $Using:_
            Result  = Invoke-Pester -Configuration $pc
        }
    } @splatStartThreadJob
}

Wait-Job -Job $threadJobs | Receive-Job

3. If there was no error on first try, then start new PowerShell session, do not use existing session

Output:

Describe your environment

Pester version : 5.5.0 C:\Program Files\WindowsPowerShell\Modules\Pester\5.5.0\Pester.psm1
PowerShell version : 7.3.6
OS version : Microsoft Windows NT 10.0.19045.0

Possible Solution?

Do not use the -StreamingHost parameter in Start-ThreadJob when calling Invoke-Pester

OR:

Replace line 13998 in Pester.psm1 with the following:

 elseif ($null -ne $host.UI -and ($hostProperties = $host.UI.psobject.Properties) -and ($supportsVT = $hostProperties['SupportsVirtualTerminal']) -and $supportsVT.Value) {

[kborowinski]

BTW, just for fun, repro code refactored to use ForEach-Object -Parallel works:

$cpuCount = [Environment]::ProcessorCount
$results = 1..($cpuCount * 2) | ForEach-Object -Parallel {
    Import-Module Pester -Force -ErrorAction Stop
    $pc = New-PesterConfiguration
    $pc.Run.Container = New-PesterContainer -ScriptBlock {Describe 'd' { It 'i' { 1 | Should -Be 1 }}}
    $pc.Output.Verbosity = 'None'
    $pc.Run.PassThru = $true
    $pc.Run.SkipRemainingOnFailure ='Block'
    [PSCustomObject]@{
        Test    = $_
        Result  = Invoke-Pester -Configuration $pc
    }
} -ThrottleLimit $cpuCount
$results

Output:

The issue is - this is for PowerShell 7+ only. I need parallelization on PowerShell 5.1. Also, @PaulHigin it looks like the $Host.UI.psobject.Properties behavior is ThreadJob bug or limitation?

[fflaten]

Did you check if UI is null? psobject.properties might work silently because of member enumeration. Not in front of a computer today so can't test myself

Does it work without -StreamingHost?

[kborowinski]

It does work without StreamingHost 👀:

$Host.UI is not null but $Host.UI.psobject.Properties are:

This has to be ThreadJob bug or limitation that I didn't know about 🤔

[kborowinski]

This is very strange:

1. I've changed the Pester.psm1 module to wait for debugger when $Host.UI.posobject.Properties are $null (and store the value in the $p variable):

2. Then I've attached to the process and entered the runspace for debugging:

And as you can see in the right window the $p variable is really $null but by the time the debugger was attached to the runspace the $Host.UI.psobject.Properties has all the values and is not null anymore 👀👀👀

[kborowinski]

OK, got it! Finally! The $Host.UI is kind of lazy loaded! (possible race condition)

I have modified the Pester.psm1 once more with simple loop that waits until the $Host.UI.psobject.Properties are available:

And it works now - this is the output of the repro script now:

Really strange behavior. I would love if @SeeminglyScience could have a look at it for some thorough explanation.

[kborowinski]

@fflaten I was able to chat with @SeeminglyScience yesterday on Discord, and for him it looks like race condition bug (I hope he doesn't mind me posting the conversation screenshot here):

I will investigate some more if that's PowerShell or ThreadJob module bug or something else. In the meantime, I'll not use StreamingHost in my parallel testing module since, in that case, I do not need it anyways.

[kborowinski]

This is yet simplest repro case (the number of jobs has to be quite high though to get the repro):

1..500 | % {Start-ThreadJob {1 | Should -BeExactly 1; $supportsVT = $Host.UI.psobject.Properties['SupportsVirtualTerminal'].Value} -StreamingHost $Host -InitializationScript {Import-Module Pester -Force}} | Wait-Job | Receive-Job | Remove-Job -Force

Output:

Error:

[fflaten]

Thank you for the thorough troubleshooting! This is a weird one and a race-condition sounds right.

I was able to reproduce a couple of times then suddenly not, so wasn't able to see if we could make the repro simpler. I doubt module import and calling Should is relevant, so was hoping to reproduce with sleep in scriptblock and/or initializationscript ++.

Either way I support the suggested fix with a few conditions:

• Bug report is created in repo for ThreadJob or PowerShell
• Issue is referenced in comment above the fix

Reason being that checking psobject.properties and not just $host.UI is unexpected and would likely be removed in a future cleanup/refactor PR + hopefully they're able to fix the root cause which might help others as well 🙂

Do you want to provide the PR once the bug report is created by you or Patrick?

[kborowinski]

I was able to reproduce the issue without InitializationScript but I still needed to call Should. I was unable to reproduce the issue otherwise.

I also think that since this is race condition then checking only psobject.Properties is not the right solution, because the exception could happen on $Host.UI for someone else, therefore I would refrain from implementing my current workaround.

I think that for the moment there are 2 options:

1. I could suggest in the workaround section to not use StreamingHost when calling Pester from Start-ThreadJob
2. PR with a full $Host.UI check:

 elseif ($null -ne $host.UI -and ($hostProperties = $host.UI.psobject.Properties) -and ($supportsVT = $hostProperties['SupportsVirtualTerminal']) -and $supportsVT.Value) {

What do you think @fflaten?

Ps. I will create an issue in the ThreadJob module repo though.

[kilasuit]

@kborowinski - In your original testing when using Start-ThreadJob I see that you are using the older of the two ThreadJob modules. In my testing in 7.3.6 & 5.1 on Microsoft Windows 10.0.25905 - aka Win 11 23H2 via the Insiders Dev ring with the newer Microsoft.PowerShell.ThreadJob module v2.1.0 I did not get it throwing that error, using this to test 1..500 | % {Start-ThreadJob {1 | Should -BeExactly 1; $supportsVT = $Host.UI.psobject.Properties['SupportsVirtualTerminal'].Value} } | Wait-Job | Receive-Job

So this may be fixed for you using that module (which due to command ordering preferences is being autoloaded over the ThreadJob Module for me) so can you please test this out for me?

Also this wasn't tested using Windows Terminal & I am just about to retest there too.

Additional background info
ThreadJob (v2.0.3) as is what is currently included in 7.3.6 however in future (somepoint soon as I have a PR for this) Microsoft.PowerShell.ThreadJob (v2.1.0 - and is the successor to ThreadJob going forward - joys of module renames eh!!) will also be included. I also noted that you mention that you need this to work on PowerShell 5.1 as well as in 7.x.x - does testing with Microsoft.PowerShell.ThreadJob (v2.1.0) on both repro for you too?
FYI Paul Higin has recently retired so may not answer the question you posted to him above & also thanks for rasing this issue in the PowerShell Org ThreadJob Repo too though as above I think this possibly may have been fixed.

[kborowinski]

@kilasuit Unfortunately the same issue occurs in Microsoft.PowerShell.ThreadJob v2.1.0

[fflaten]

2. PR with a full $Host.UI check:

 elseif ($null -ne $host.UI -and ($hostProperties = $host.UI.psobject.Properties) -and ($supportsVT = $hostProperties['SupportsVirtualTerminal']) -and $supportsVT.Value) {

I'm fine with this workaround incl. a comment with link to this issue since I doubt we'll be able to cover this with a test consistently.

Thanks for really troubleshooting this and sorry for the delayed response 🙂 . Would you still like to provide a PR?

[kborowinski]

No problem, I will provide a PR.