Pester returns errors even when no tests fail

[cjfarrelly]

Where are you running it?

• Azure DevOps Service (VSTS)

Version of Extension/Task

10.0.5

Expected behaviour and actual behaviour

Expected: Pester task passes without error when no tests fail

Actual: Pester task fails with error even though all tests pass

Steps to reproduce the problem

Here's a link to a public project where this issue can be repro'd

https://dev.azure.com/cfarrelly/test/_build/results?buildId=4&view=logs&j=275f1d19-1bd8-5591-b06b-07d489ea915a&t=f969aa47-2434-555b-2817-6587663ed32a

I thought originally it may be down to tags and skipping tests but I created a simple repro here with 1 test to run and the test runner fails with 'Pester returned errors', appears to be an issue when checking the results here

AzureDevOpsExtension/Extension/PesterTask/PesterV10/Pester.ps1

Line 145 in e2141f2

if ($result.failedCount -ne 0) {

Thanks,

Colin.

[ChrisLGardner]

Thanks for reporting this. I'll have a look at what's going on here and see if I can fix it this week.

[fmad]

Just got bitten by this one too: was starting to add some tests and couldn't figure out why they were passing on my local machine and even showing as passing in the output logs but then the task fails with:

D:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1
D:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1
+ d:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
 
+ d:\a\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1

[ChrisLGardner]

Thanks for the extra reports. I'll bump this up my list and see if I can get it done tonight or earlier.

[dkalbertson]

Just hit this bug as well...

2020-06-03T12:15:39.5996504Z Tests Passed: 3, 
2020-06-03T12:15:39.6033071Z Failed: 0, 
2020-06-03T12:15:39.6070344Z Skipped: 0 
2020-06-03T12:15:39.6150264Z NotRun: 0
2020-06-03T12:15:39.6180214Z 
2020-06-03T12:15:39.6181136Z 
2020-06-03T12:15:40.2186133Z ##[error]C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1

2020-06-03T12:15:40.2198914Z ##[error]C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1

2020-06-03T12:15:40.2217307Z ##[error]+ C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
 

2020-06-03T12:15:40.2220096Z ##[error]+ C:\Build\CI-Agent2\_work\_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1
 

2020-06-03T12:15:40.2716171Z Pester Script finished
2020-06-03T12:15:40.2875525Z ##[section]Finishing: Pester Test Runner

[Indrakanth-star]

I too faced the same error and waiting for the solution. Please find below for the raw error log report.

Also the test result is showing as passed even though I intentionally placed the code to fail the test.

2020-06-04T10:22:55.0696725Z ##[error]D:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1

• d:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...

2020-06-04T10:22:55.0709967Z ##[error]D:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester.ps1 : Pester returned errors
At line:1 char:1

• d:\a_tasks\Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9\10.0.5\Pester ...

2020-06-04T10:22:55.0729293Z ##[error]+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1

2020-06-04T10:22:55.0731667Z ##[error]+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Pester.ps1

2020-06-04T10:22:55.1061073Z Pester Script finished
2020-06-04T10:22:55.1140399Z ##[section]Finishing: Pester Test Runner

[ChrisLGardner]

Thanks for the extra reports, I'm working on a fix but I also want to improve my testing in the pipeline to make sure I catch more of these issues before they hit users. It's going to take me a few hours of work to do that so I'm hoping to get it done this weekend.

[ChrisLGardner]

This should be fixed now in 10.1.x, took a bit longer than expected as my testing pipeline needed work and then wasn't quite doing what it should have.

[Indrakanth-star]

Hi @ChrisLGardner
Thanks for resolving the errors.
Still i am able to see the issue as below
"test result is showing as passed even though I intentionally placed the code to fail the test"

Can you please look into this too.

Thanks
Indra

[ChrisLGardner]

Hi @ChrisLGardner
Thanks for resolving the errors.
Still i am able to see the issue as below
"test result is showing as passed even though I intentionally placed the code to fail the test"

Can you please look into this too.

Thanks
Indra

I saw this a few times when I was fixing the task. What version is it trying to run? If you look in the logs for the task it'll give you an exact version number at the top.

[Indrakanth-star]

Task : Pester Test Runner
Description : Run Pester tests by either installing the latest version of Pester at run time (if possible) or using the version shipped with the task (5.0.0)
Version : 10.1.12
Author : Pester
Help : Version: #{Build.BuildNumber}#. More Information

[Indrakanth-star]

Hi @ChrisLGardner , Any update on the issue ?

[jhampson-dbre]

It looks like the Pester5 result object has changed significantly since v4 and no longer has a failedcount property. Helper command ConvertTo-Pester4Result is available to transform a v5 result object to v4 format.

[thisjustin816]

Also, what do we do if we need to use Pester v4? Since v10 of the task with this fix doesn't support it.

I can't upgrade all my tests yet because of the bug in Pester v5 where variables in BeforeAll are only set on the second run of the tests.

[ChrisLGardner]

You can use v9 of the task as that supports v4 of Pester.

I didn't get a chance to look at this at the weekend but I'll try to find some time this week.

[thisjustin816]

You can use v9 of the task as that supports v4 of Pester.

I didn't get a chance to look at this at the weekend but I'll try to find some time this week.

I mean once the fix is put out, unless you're planning to update v9 of the task as well (I'm not familiar with the task publishing process so not sure if that is possible).

[ChrisLGardner]

You can use v9 of the task as that supports v4 of Pester.
I didn't get a chance to look at this at the weekend but I'll try to find some time this week.

I mean once the fix is put out, unless you're planning to update v9 of the task as well (I'm not familiar with the task publishing process so not sure if that is possible).

My plan is to maintain v9 of the task as long as Pester 4 is in use, it hopefully doesn't need many fixes as it's had plenty of time to get those so far but anything we find in v10 that is an issue in v9 will get fixed in both.

[ChrisLGardner]

I've added another fix that should hopefully help with this issue if people are still seeing problems.