Azure DevOps "Publish test results" task can't recognize tests if BeforeAll throws exception

[antonGritsenko]

1. General summary of the issue

When exception throw in the BeforeAll, the tests are not recognized by the Publish Test Result task in Azure DevOps

Pester executed as simple PowerShell script:

Install-Module Pester -Force -SkipPublisherCheck 
Install-Module Microsoft.Graph.Intune -Force -SkipPublisherCheck 
Import-Module Pester
Import-Module ".\DummyIntuneModule.psm1" -Force
Invoke-Pester -OutputFile "$(System.DefaultWorkingDirectory)\Test-Pester.XML" -OutputFormat NUnitXML

The Publish test results task configured to NUnit format.

2. Describe Your Environment

Pester version : 5.0.4 C:\Program Files\WindowsPowerShell\Modules\Pester\5.0.4\Pester.psd1
PowerShell version : 5.1.14393.3866
OS version : Microsoft Windows NT 10.0.14393.0
Publish test results to Azure Pipelines task version: 2.171.0

3. Expected Behavior

The tests should be recognized by the Publish test results to Azure Pipelines tasks as failed.

4.Current Behavior

The XML generated by the Pester looks like correct (see bottom of the issue), but can't be recognized by the Publish test results to Azure Pipelines (version 2.171.0):

No Result Found to Publish 'D:\a\1\s\Test-Pester.XML'.
No build level attachments to publish.
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore

If one of the test failed then tests are correctly recognized by the task.

5. Possible Solution

No

6. Context

If BeforeAll failed, then all tests must be failed. At the moment they all are "green", this breaks behavior.

Result XML:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
 <test-results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="nunit_schema_2.5.xsd" name="Pester" total="9" errors="0" failures="9" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2020-10-13" time="14:27:10">
   <environment clr-version="4.0.30319.42000" user-domain="fv-az73" cwd="D:\a\1\s\TestProject.PoCs\TestProject.PoC2" platform="Microsoft Windows Server 2016 Datacenter|C:\windows|\Device\Harddisk0\Partition2" machine-name="fv-az73" nunit-version="2.5.8.0" os-version="10.0.14393" user="VssAdministrator" />
   <culture-info current-culture="en-US" current-uiculture="en-US" />
   <test-suite type="TestFixture" name="Pester" executed="True" result="Failure" success="False" time="4.337" asserts="0" description="Pester">
     <results>
       <test-suite type="TestFixture" name="D:\a\1\s\TestProject.PoCs\TestProject.PoC2\SwpIntuneModule.Tests.ps1" executed="True" result="Failure" success="False" time="4.337" asserts="0" description="D:\a\1\s\TestProject.PoCs\TestProject.PoC2\SwpIntuneModule.Tests.ps1">
         <results>
           <test-suite type="TestFixture" name="SwpIntubeDeviceConfiguration " executed="True" result="Failure" success="False" time="1.6834" asserts="0" description="SwpIntubeDeviceConfiguration ">
             <results>
               <test-suite type="TestFixture" name="SwpIntubeDeviceConfiguration .when wrong parameter passed" executed="True" result="Failure" success="False" time="0" asserts="0" description="SwpIntubeDeviceConfiguration .when wrong parameter passed">
                 <results>
                   <test-case description="" name="" time="0" asserts="0" success="False" result="Failure" executed="True">
                     <failure>
                       <message>This test should run but it did not. Most likely a setup in some parent block failed.</message>
                       <stack-trace />
                     </failure>
                   </test-case>
                   <test-case description="" name="" time="0" asserts="0" success="False" result="Failure" executed="True">
                     <failure>
                       <message>This test should run but it did not. Most likely a setup in some parent block failed.</message>
                       <stack-trace />
                     </failure>
                   </test-case>
                   <test-case description="" name="" time="0" asserts="0" success="False" result="Failure" executed="True">
                     <failure>
                       <message>This test should run but it did not. Most likely a setup in some parent block failed.</message>
                       <stack-trace />
                     </failure>
                   </test-case>
                 </results>
               </test-suite>
               <test-suite type="TestFixture" name="DmIntubeDeviceConfiguration .when nothing created in the Intune" executed="True" result="Failure" success="False" time="0" asserts="0" description="DmIntubeDeviceConfiguration .when nothing created in the Intune">
                 <results>
                   <test-case description="" name="" time="0" asserts="0" success="False" result="Failure" executed="True">
                     <failure>
                       <message>This test should run but it did not. Most likely a setup in some parent block failed.</message>
                       <stack-trace />
                     </failure>
                   </test-case>
                   <test-case description="" name="" time="0" asserts="0" success="False" result="Failure" executed="True">
                     <failure>
                       <message>This test should run but it did not. Most likely a setup in some parent block failed.</message>
                       <stack-trace />
                     </failure>
                   </test-case>
                   <test-case description="" name="" time="0" asserts="0" success="False" result="Failure" executed="True">
                     <failure>
                       <message>This test should run but it did not. Most likely a setup in some parent block failed.</message>
                       <stack-trace />
                     </failure>
                   </test-case>
                 </results>
               </test-suite>
               <test-suite type="TestFixture" name="DmIntubeDeviceConfiguration .when config already in the Intune" executed="True" result="Failure" success="False" time="0" asserts="0" description="DmIntubeDeviceConfiguration .when config already in the Intune">
                 <results>
                   <test-case description="" name="" time="0" asserts="0" success="False" result="Failure" executed="True">
                     <failure>
                       <message>This test should run but it did not. Most likely a setup in some parent block failed.</message>
                       <stack-trace />
                     </failure>
                   </test-case>
                   <test-case description="" name="" time="0" asserts="0" success="False" result="Failure" executed="True">
                     <failure>
                       <message>This test should run but it did not. Most likely a setup in some parent block failed.</message>
                       <stack-trace />
                     </failure>
                   </test-case>
                   <test-case description="" name="" time="0" asserts="0" success="False" result="Failure" executed="True">
                     <failure>
                       <message>This test should run but it did not. Most likely a setup in some parent block failed.</message>
                       <stack-trace />
                     </failure>
                   </test-case>
                 </results>
               </test-suite>
             </results>
           </test-suite>
         </results>
       </test-suite>
     </results>
   </test-suite>
 </test-results>

[antonGritsenko]

Seems to be caused by the empty name attribute of the <test-case> element

[antonGritsenko]

@nohwnd thanks a lot! nice job. Any chance that it will be released as patch for 5.0?

[nohwnd]

I'd rather avoid that.