Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Alter amount of tests that run per PR for mono mobile configurations #60727

Merged
merged 16 commits into from
Oct 29, 2021
Merged

Alter amount of tests that run per PR for mono mobile configurations #60727

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a8bb9f1
Alter amount of tests that run per PR for mono mobile configurations
Oct 21, 2021
5e47dda
Missing tab on condition
Oct 21, 2021
45516d9
Merge remote-tracking branch 'upstream/main' into full-tests-on-rolling
Oct 27, 2021
e37e411
See if the yml is better
Oct 27, 2021
ea4afbf
Variable in wrong place
Oct 27, 2021
7b2652d
yml :-(
Oct 27, 2021
2649f5d
silly checks
Oct 27, 2021
67e444a
yml syntax
Oct 27, 2021
3415612
Rollback runtime.yml and see why it's not triggering
Oct 28, 2021
7c3c210
Add smoke test args back in
Oct 28, 2021
218a770
Add condition back in
Oct 28, 2021
5b203e1
Merge remote-tracking branch 'upstream/main' into full-tests-on-rolling
Oct 28, 2021
71a7521
Even more conditions
Oct 28, 2021
01d5578
Remove echo middleware for wasm when smoke tests are run
Oct 28, 2021
d01b7b5
Merge remote-tracking branch 'upstream/main' into full-tests-on-rolling
Oct 29, 2021
3956da7
Exclude more runtime test PR runs. PR feedback
Oct 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions eng/pipelines/libraries/base-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ jobs:
- _extraHelixArguments: ''
- _crossBuildPropertyArg: ''
- _testRunNamePrefixSuffix: ''
- _runSmokeTestsOnlyArg: ''
steveisok marked this conversation as resolved.
Show resolved Hide resolved

- librariesBuildArtifactName: ${{ format('libraries_bin_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig) }}

Expand Down Expand Up @@ -87,6 +88,9 @@ jobs:
- ${{ if eq(parameters.isOfficialBuild, 'true') }}:
- _msbuildCommonParameters: /p:OfficialBuildId=$(Build.BuildNumber)

- ${{ if eq(variables['isFullMatrix'], false) }}:
- _runSmokeTestsOnlyArg: /p:RunSmokeTestsOnly=true

- _runtimeArtifactName: ''
- _runtimeDownloadPath: ''
- _runtimeArtifactsPathArg: ''
Expand Down
69 changes: 35 additions & 34 deletions eng/pipelines/runtime-staging.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ jobs:
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_Mono
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg)
timeoutInMinutes: 180
condition: >-
or(
Expand Down Expand Up @@ -249,16 +249,15 @@ jobs:
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_Mono
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg)
timeoutInMinutes: 180
condition: >-
or(
eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true),
eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true),
eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true),
eq(variables['isFullMatrix'], true))

# don't run tests on PRs until we can get significantly more devices
# Device tests are rolling build only
${{ if eq(variables['isFullMatrix'], true) }}:
# extra steps, run tests
extraStepsTemplate: /eng/pipelines/libraries/helix.yml
Expand Down Expand Up @@ -320,7 +319,7 @@ jobs:
buildConfig: Release
runtimeFlavor: mono
platforms:
- Android_x64
- Android_x64
variables:
- ${{ if and(eq(variables['System.TeamProject'], 'public'), eq(variables['Build.Reason'], 'PullRequest')) }}:
- name: _HelixSource
Expand All @@ -343,11 +342,13 @@ jobs:
eq(dependencies.evaluate_paths.outputs['SetPathVars_runtimetests.containsChange'], true),
eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true),
eq(variables['isFullMatrix'], true))
# extra steps, run tests
extraStepsTemplate: /eng/pipelines/common/templates/runtimes/android-runtime-and-send-to-helix.yml
extraStepsParameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)
# NOTE: Per PR test execution is not recommended for runtime tests
${{ if eq(variables['isFullMatrix'], true) }}:
# extra steps, run tests
extraStepsTemplate: /eng/pipelines/common/templates/runtimes/android-runtime-and-send-to-helix.yml
extraStepsParameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Build the whole product using Mono and run runtime tests with the JIT.
Expand All @@ -359,7 +360,7 @@ jobs:
buildConfig: Release
runtimeFlavor: mono
platforms:
- iOSSimulator_x64
- iOSSimulator_x64
variables:
- ${{ if and(eq(variables['System.TeamProject'], 'public'), eq(variables['Build.Reason'], 'PullRequest')) }}:
- name: _HelixSource
Expand All @@ -381,15 +382,14 @@ jobs:
eq(dependencies.evaluate_paths.outputs['SetPathVars_runtimetests.containsChange'], true),
eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true),
eq(variables['isFullMatrix'], true))
# Test execution is temporarily disabled because test apps no longer launch
# and the test suite times out after two hours, even if xharness cannot
# successfully launch any tests. Re-enable once these issues have been fixed.
# NOTE: Per PR test execution is not recommended for runtime tests
#
# extra steps, run tests
# extraStepsTemplate: /eng/pipelines/common/templates/runtimes/android-runtime-and-send-to-helix.yml
# extraStepsParameters:
# creator: dotnet-bot
# testRunNamePrefixSuffix: Mono_$(_BuildConfig)
# ${{ if eq(variables['isFullMatrix'], true) }}:
# extra steps, run tests
# extraStepsTemplate: /eng/pipelines/common/templates/runtimes/android-runtime-and-send-to-helix.yml
# extraStepsParameters:
# creator: dotnet-bot
# testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Build the whole product using Mono for Android and run runtime tests with Android devices
Expand All @@ -401,7 +401,7 @@ jobs:
buildConfig: Release
runtimeFlavor: mono
platforms:
- Android_arm64
- Android_arm64
variables:
- ${{ if and(eq(variables['System.TeamProject'], 'public'), eq(variables['Build.Reason'], 'PullRequest')) }}:
- name: _HelixSource
Expand Down Expand Up @@ -460,20 +460,21 @@ jobs:
eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true),
eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true),
eq(variables['isFullMatrix'], true))
# extra steps, run tests
extraStepsTemplate: /eng/pipelines/libraries/helix.yml
extraStepsParameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)
extraHelixArguments: /p:BrowserHost=windows
scenarios:
- normal
- wasmtestonbrowser
condition: >-
or(
eq(variables['librariesContainsChange'], true),
eq(variables['monoContainsChange'], true),
eq(variables['isFullMatrix'], true))
${{ if eq(variables['isFullMatrix'], true) }}:
# extra steps, run tests
extraStepsTemplate: /eng/pipelines/libraries/helix.yml
extraStepsParameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)
extraHelixArguments: /p:BrowserHost=windows
scenarios:
- normal
- wasmtestonbrowser
condition: >-
or(
eq(variables['librariesContainsChange'], true),
eq(variables['monoContainsChange'], true),
eq(variables['isFullMatrix'], true))

#
# CoreCLR Build for running Apple Silicon libraries-innerloop
Expand Down
34 changes: 20 additions & 14 deletions eng/pipelines/runtime.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,7 @@ jobs:
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_Mono
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg)
timeoutInMinutes: 180
condition: >-
or(
Expand Down Expand Up @@ -323,7 +323,9 @@ jobs:
buildConfig: Release
runtimeFlavor: mono
platforms:
- Browser_wasm
# BuildWasmApps should only happen on the rolling build. No need to duplicate the build on PR's
${{ if eq(variables['isFullMatrix'], true) }}:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe instead of this, we can have tighter constraints on which changes trigger running Wasm.Build.Tests? Eg. src/mono, src/tests/BuildWasmApps, src/tasks, etc? though it will get kinda detailed, because we would want to include the various nugets consumed for workloads.

Just fyi, this job does not run the library tests as we pass TestAssemblies=false, so it builds, and then runs only the Wasm.Build.Tests .
cc @lewing

Copy link
Member

@lewing lewing Oct 21, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather drop Scenario='normal' from Browser_wasm and keep EAT with Scenario='normal', I think this is actually a little beyond guidance and it would catch a lot of test configuation changes that would otherwise cause us problems.

- Browser_wasm
variables:
# map dependencies variables to local variables
- name: monoContainsChange
Expand Down Expand Up @@ -373,7 +375,7 @@ jobs:
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_Mono_EAT
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=false
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=false
timeoutInMinutes: 180
condition: >-
or(
Expand Down Expand Up @@ -415,7 +417,7 @@ jobs:
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_Mono_AOT
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=true
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) $(_runSmokeTestsOnlyArg) /p:ArchiveTests=true /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't want to run EAT/AOT tests for every PR, then it would be useful to have some way to opt-in, or trigger these builds on a PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we need a way to trigger aot manually for sure, it would be nice to auto trigger it with any src/mono/mono change as well.

timeoutInMinutes: 180
condition: >-
or(
Expand Down Expand Up @@ -491,11 +493,13 @@ jobs:
eq(dependencies.evaluate_paths.outputs['SetPathVars_runtimetests.containsChange'], true),
eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true),
eq(variables['isFullMatrix'], true))
# extra steps, run tests
extraStepsTemplate: /eng/pipelines/common/templates/runtimes/wasm-runtime-and-send-to-helix.yml
extraStepsParameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)
# NOTE: Per PR test execution is not recommended for mobile runtime tests
${{ if eq(variables['isFullMatrix'], true) }}:
# extra steps, run tests
extraStepsTemplate: /eng/pipelines/common/templates/runtimes/wasm-runtime-and-send-to-helix.yml
extraStepsParameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Build the whole product using Mono for Android and run runtime tests with Android emulator
Expand Down Expand Up @@ -529,11 +533,13 @@ jobs:
eq(dependencies.evaluate_paths.outputs['SetPathVars_runtimetests.containsChange'], true),
eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true),
eq(variables['isFullMatrix'], true))
# extra steps, run tests
extraStepsTemplate: /eng/pipelines/common/templates/runtimes/android-runtime-and-send-to-helix.yml
extraStepsParameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)
# NOTE: Per PR test execution is not recommended for mobile runtime tests
${{ if eq(variables['isFullMatrix'], true) }}:
# extra steps, run tests
extraStepsTemplate: /eng/pipelines/common/templates/runtimes/android-runtime-and-send-to-helix.yml
extraStepsParameters:
creator: dotnet-bot
testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Build Mono and Installer on LLVMJIT mode
Expand Down
10 changes: 7 additions & 3 deletions src/libraries/tests.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,11 @@
<ProjectExclusions Remove="$(MSBuildThisFileDirectory)System.IO.IsolatedStorage\tests\System.IO.IsolatedStorage.Tests.csproj" />
</ItemGroup>

<ItemGroup>
<ItemGroup Condition="'$(RunSmokeTestsOnly)' == 'true'">
<ProjectReference Include="$(RepoRoot)\src\libraries\System.Runtime\tests\System.Runtime.Tests.csproj" />
steveisok marked this conversation as resolved.
Show resolved Hide resolved
</ItemGroup>

<ItemGroup Condition="'$(RunSmokeTestsOnly)' != 'true'">
<ProjectReference Include="$(MSBuildThisFileDirectory)*\tests\**\*.Tests.csproj"
Exclude="@(ProjectExclusions)"
Condition="'$(TestAssemblies)' == 'true'" />
Expand Down Expand Up @@ -344,7 +348,7 @@
BuildInParallel="false" />
</ItemGroup>

<ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(TargetOS)' == 'Android'">
<ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(RunSmokeTestsOnly)' != 'true' and '$(TargetOS)' == 'Android'">
<ProjectReference Include="$(MonoProjectRoot)sample\Android\AndroidSampleApp.csproj"
BuildInParallel="false" />
<ProjectReference Include="$(RepoRoot)\src\tests\FunctionalTests\Android\**\*.Test.csproj"
Expand All @@ -353,7 +357,7 @@
</ItemGroup>

<!-- Don't build samples, and functional tests on EAT, AOT, and WBT lanes -->
<ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(TargetOS)' == 'Browser' and '$(BuildAOTTestsOnHelix)' != 'true' and '$(TestWasmBuildTests)' != 'true'">
<ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(RunSmokeTestsOnly)' != 'true' and '$(TargetOS)' == 'Browser' and '$(BuildAOTTestsOnHelix)' != 'true' and '$(TestWasmBuildTests)' != 'true'">
<ProjectReference Include="$(MonoProjectRoot)sample\wasm\**\*.Sample.csproj"
Exclude="@(ProjectExclusions)"
BuildInParallel="true" />
Expand Down