Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor WinUI Helix functionality into reusable WinUI.Helix nuget package #4678

Merged
merged 32 commits into from
Apr 20, 2021

Conversation

kmahone
Copy link
Member

@kmahone kmahone commented Mar 29, 2021

Since starting to use Helix to execute WinUI tests, we have built up a good amount of scaffolding to sit on top of Helix that builds on its out-of-the-box functionality.

This code is generally useful to other teams who want to use Helix to run TAEF tests. The WinUI 2 and WinUI 3 repos both have copies of this code, but it has diverged over time. Also, other teams outside of WinUI have made forks of this code (such as Terminal).

So, I refactored the code in the WinUI3 repo to separate out the Xaml specific code from the general purpose code. The general purpose code is packaged into a nuget package (Microsoft.Internal.WinUI.Helix).

The copy of the code in this repo is removed and we use the nuget package instead.

WinUI.Helix Features

TAEF Support

Out of the box, Helix has support for XUnit which is the test runner that the dotnet team use to run
their tests. WinUI required TAEF support which was implemented on top of Helix's base functionality. There are two parts
to the TAEF support:

TAEF Test Execution

RunHelixWorkItem.ps1 from WinUI.Helix invokes TAEF. It then converts TAEF's .wtl log output to the XUnit log format that
Helix supports. This allows us to execute TAEF tests in Helix and get the test results reported back to the Azure Pipeline.

TAEF Test Discovery

It is possible to hard-code the set of Helix Work Items to run. But this can be fragile when you have a large amount of
tests. GenerateHelixWorkItems.ps1 from WinUI.Helix is a script that can be run from Azure Pipelines to discover TAEF tests
and generate the work items for Helix.

Retry Logic

In the ideal world, tests would be 100% reliable and would never fail except when catching a legitimate error. However, sometimes it is necessary to handle a certain amount of unreliability in a suite of tests. WinUI.Helix includes some retry logic. A test that fails is initially re-run and is only reported as a true failure if it fails multiple times on re-run.

Currently this retry-logic is always enabled. But it could be made opt-out if that was needed.

Test Output as Pipeline Artifact

To aid debugging test failures, the full log output of the executed tests along with any supporting files (e.g. screenshots)
is uploaded as an Azure Pipeline artifact. This is done by WinUI.Helix's ProcessHelixFiles.ps1.

Extension Points

If you need to run some custom logic, but still want to use WinUI.Helix's TAEF execution, you can provide scripts that
will be called at various points in the execution. This is useful if you need to install some dependencies on the test
machine before executing the tests.

@ghost ghost added the needs-triage Issue needs to be triaged by the area owners label Mar 29, 2021
@kmahone kmahone changed the title FHL: Refactor WinUI Helix functionality into reusable WinUI.Helix nuget package Refactor WinUI Helix functionality into reusable WinUI.Helix nuget package Mar 29, 2021
@kmahone kmahone requested review from llongley and ranjeshj March 29, 2021 22:08
@StephenLPeters StephenLPeters added area-TestInfrastructure Issue in the test infrastructure (e.g. in Helix scripts) team-Controls Issue for the Controls team and removed needs-triage Issue needs to be triaged by the area owners labels Mar 29, 2021
@kmahone kmahone requested a review from StephenLPeters April 6, 2021 17:24
Copy link
Contributor

@StephenLPeters StephenLPeters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@StephenLPeters
Copy link
Contributor

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@kmahone
Copy link
Member Author

kmahone commented Apr 20, 2021

/azp run

@azure-pipelines
Copy link

Pull request contains merge conflicts.

@kmahone
Copy link
Member Author

kmahone commented Apr 20, 2021

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@kmahone kmahone merged commit 5b757e8 into master Apr 20, 2021
@kmahone kmahone deleted the user/kmahone/helixpartypack branch April 20, 2021 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-TestInfrastructure Issue in the test infrastructure (e.g. in Helix scripts) team-Controls Issue for the Controls team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants