Provide JUnit output for PIO Unit Testing

[tschissler]

I was able to create a set of Unit Tests with PlatformIO and run them as native within a docker container. I also managed to run those tests as port of my build pipeline on Azure DevOps. All looks pretty good.

Thank you so much for building this amazing tool!!!

Only thing which nags me is that failing unit tests make my build fail with no additional information available before I dig into the log files.
It would be amazing, if I could extract the build results into a test result file in a JUnit format or something alike which I then could use to provide my build system with additional information.
I'm wondering if something like this is build in right now or if I have to build something on my own.

Useful links

• https://llg.cubic.org/docs/junit/
• https://help.catchsoftware.com/display/ET/JUnit+Format

[ivankravets]

Do you mean pio test -v?

[tschissler]

Not really. That would only add more details, if I understand this option correctly.
I have solved the issue by writing the console output to a file and then parsing this to create the XML format used by NUnit and others.
I'll publish the python script I created and link it to this thread within the next couple of days and then close it.

[ivankravets]

Maybe pio test —json-output will help here?

[ivankravets]

Where is the specification for this format in XML?

[tschissler]

@ivankravets I do not know the pio test --json-output option and on my version it seems not to be supported nor can I find documentation about it. That would certainly be helpful as JSON would be easier to parse than the text output.

The XML format I am using is this here: https://github.com/nunit/docs/wiki/Test-Result-XML-Format. But there would be other options possible as my goal is to use this task in a Azure DevOps build pipeline: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml to publish my test results and make them available to be seen on the build report. The result then looks like this:

I have shared the Python script I have build here: https://github.com/tschissler/sharing/tree/master/Scripts
Bare with me as I am a Python newbie :-) But maybe someone else might find this helpful.

I'm now working on an article describing how to integrate Platformio with Azure DevOps CI/CD. Maybe you want to consider adding this option to your documentation of CI/CD systems.

[ivankravets]

I renamed this issue to "Provide JSON output for PIO Unit Testing". So we will implement it soon.

[jpsfs]

Hi!
I'm also looking for the same kind of functionality. Is there any idea when will this be available?

@tschissler could you share your yml pipeline?

[ivankravets]

Please vote for this feature here #2891 (comment) 👍

See https://github.com/platformio/platformio-core/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc

[tschissler]

I'll share my solution later. Working on a Hackton against the virus this weekend.

[tschissler]

@jpsfs This is my YAML file for the build. I use a docker container where the build runs in.
Let me know if you have any questions.

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

resources:
  containers:
  - container: platformio
    image: infinitecoding/platformio-for-ci:latest   
    options: -u root    

trigger:
- master

jobs:
- job: esp32_platformio
  displayName: "PlatformIO build"
  container: platformio

  pool:
    vmImage: 'ubuntu-latest'

  steps: 
    - script: dir -a
      displayName: "Show current directory content"
    - script: cd battleship-embedded/BattleShip
      displayName: "Open Battleship project directory"
    - script: platformio test -e native -d battleship-embedded/BattleShip >> testresults.txt
      displayName: "Run Unit Tests"
      continueOnError: true
    - script: python $(Build.SourcesDirectory)/battleship-embedded/BattleShip/_deploy/TestResultsParser.py testresults.txt testresults.xml
      displayName: "Converting Test Results"
    - task: PublishTestResults@2
      inputs:
        testResultsFormat: 'NUnit'
        testResultsFiles: 'testresults.xml'
        failTaskOnFailedTests: true
    - script: platformio run -e esp32 -d battleship-embedded/BattleShip
      displayName: "Build for ESP32 platform"
    - task: CopyFiles@2
      inputs:
        SourceFolder: $(Build.SourcesDirectory)/battleship-embedded/BattleShip/.pio/build/esp32/
        Contents: '*.bin'
        TargetFolder: $(Build.ArtifactStagingDirectory)
      displayName: "Copy build output files to ArtifactsStagingDirectory"
    - task: CopyFiles@2
      inputs:
        SourceFolder: $(Build.SourcesDirectory)/battleship-embedded/BattleShip/_deploy
        Contents: '*.ps1'
        TargetFolder: $(Build.ArtifactStagingDirectory)
      displayName: "Copy deployment scripts to ArtifactsStagingDirectory"
    - task: PublishBuildArtifacts@1
      inputs:
        ArtifactName: 'Firmware_$(Build.BuildNumber)'
        PathtoPublish: $(Build.ArtifactStagingDirectory)
        publishLocation: Container
        TargetPath: .

[tux19]

I would like to have the test results integrated into gitlab-ci. Is there already a solution to export test results without manually parsing the output?

[mickeprag]

If the output could follow some common format this would be easier to integrate into different solutions. GitLab supports JUnit files so my vote goes for this format.
https://docs.gitlab.com/ee/ci/unit_test_reports.html#how-to-set-it-up

[ivankravets]

Hi all,

The new 2 formats are implemented: JSON and JUnit. Please upgrade PlatformIO Core via pio upgrade --dev and re-test with pio test --output-junit.

Does it work for you?