-
Notifications
You must be signed in to change notification settings - Fork 703
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor WinUI Helix functionality into reusable WinUI.Helix nuget package #4678
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kmahone
changed the title
FHL: Refactor WinUI Helix functionality into reusable WinUI.Helix nuget package
Refactor WinUI Helix functionality into reusable WinUI.Helix nuget package
Mar 29, 2021
StephenLPeters
added
area-TestInfrastructure
Issue in the test infrastructure (e.g. in Helix scripts)
team-Controls
Issue for the Controls team
and removed
needs-triage
Issue needs to be triaged by the area owners
labels
Mar 29, 2021
StephenLPeters
approved these changes
Apr 13, 2021
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Pull request contains merge conflicts. |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area-TestInfrastructure
Issue in the test infrastructure (e.g. in Helix scripts)
team-Controls
Issue for the Controls team
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since starting to use Helix to execute WinUI tests, we have built up a good amount of scaffolding to sit on top of Helix that builds on its out-of-the-box functionality.
This code is generally useful to other teams who want to use Helix to run TAEF tests. The WinUI 2 and WinUI 3 repos both have copies of this code, but it has diverged over time. Also, other teams outside of WinUI have made forks of this code (such as Terminal).
So, I refactored the code in the WinUI3 repo to separate out the Xaml specific code from the general purpose code. The general purpose code is packaged into a nuget package (Microsoft.Internal.WinUI.Helix).
The copy of the code in this repo is removed and we use the nuget package instead.
WinUI.Helix Features
TAEF Support
Out of the box, Helix has support for XUnit which is the test runner that the dotnet team use to run
their tests. WinUI required TAEF support which was implemented on top of Helix's base functionality. There are two parts
to the TAEF support:
TAEF Test Execution
RunHelixWorkItem.ps1 from WinUI.Helix invokes TAEF. It then converts TAEF's .wtl log output to the XUnit log format that
Helix supports. This allows us to execute TAEF tests in Helix and get the test results reported back to the Azure Pipeline.
TAEF Test Discovery
It is possible to hard-code the set of Helix Work Items to run. But this can be fragile when you have a large amount of
tests. GenerateHelixWorkItems.ps1 from WinUI.Helix is a script that can be run from Azure Pipelines to discover TAEF tests
and generate the work items for Helix.
Retry Logic
In the ideal world, tests would be 100% reliable and would never fail except when catching a legitimate error. However, sometimes it is necessary to handle a certain amount of unreliability in a suite of tests. WinUI.Helix includes some retry logic. A test that fails is initially re-run and is only reported as a true failure if it fails multiple times on re-run.
Currently this retry-logic is always enabled. But it could be made opt-out if that was needed.
Test Output as Pipeline Artifact
To aid debugging test failures, the full log output of the executed tests along with any supporting files (e.g. screenshots)
is uploaded as an Azure Pipeline artifact. This is done by WinUI.Helix's ProcessHelixFiles.ps1.
Extension Points
If you need to run some custom logic, but still want to use WinUI.Helix's TAEF execution, you can provide scripts that
will be called at various points in the execution. This is useful if you need to install some dependencies on the test
machine before executing the tests.