Bring up full library test CI runs on iOS / tvOS devices #59503

steveisok · 2021-09-22T21:36:25Z

This change enables device runs on CI by building each test app on the helix instance it was deployed to. In past attempts, we looked at enhancing what takes place on the build machine via build tricks, compressing / cleaning up files early, etc and we could not overcome the need for excessively long timeouts and far more disk space.

The change also adopts the patterns established in the wasm test build, giving us the opportunity to support different scenario runs as well as workloads testing in the future.

Once the test runs turn green, we'll update the yml to only run the configurations on the rolling build.

See if zipping each test app first results in not running out of disk space. Changed cleanup steps after each test is built to maximize space utilization.

ghost · 2021-09-22T21:36:32Z

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

This change enables device runs on CI by building each test app on the helix instance it was deployed to. In past attempts, we looked at enhancing what takes place on the build machine via build tricks, compressing / cleaning up files early, etc and we could not overcome the need for excessively long timeouts and far more disk space.

The change also adopts the patterns established in the wasm test build, giving us the opportunity to support different scenario runs as well as workloads testing in the future.

Once the test runs turn green, we'll update the yml to only run the configurations on the rolling build.

Author:	steveisok
Assignees:	-
Labels:	`area-Infrastructure-libraries`
Milestone:	-

steveisok · 2021-09-22T21:41:12Z

What the change also supports is being able to build and run fullaot on any of the other apple configurations. MacCatalyst being the other one we want to run, but up until this point haven't been able to.

This change enables device runs on CI by building each test app on the helix instance it was deployed to. In past attempts, we looked at enhancing the build step only via build tricks, compressing / cleaning up files early, etc and we could not overcome the need for excessively long timeouts and far more disk space. The change also adopts the patterns established in the wasm test build, giving us the opportunity to support different scenario runs as well as workloads testing in the future.

eng/testing/tests.ioslike.targets

eng/testing/tests.targets

src/libraries/sendtohelix.proj

src/libraries/sendtohelixhelp.proj

MaximLipnin · 2021-09-23T08:47:35Z

src/libraries/sendtohelixhelp.proj

@@ -87,6 +91,12 @@
    <HelixPostCommand Include="taskkill.exe /f /im corerun.exe"/>
  </ItemGroup>

+  <ItemGroup Condition="'$(TargetOS)' == 'iOS' or '$(TargetOS)' == 'iOSSimulator'">


Is it for iOS only for now?

No, is good for both. Should include tvOS and tvOSSimulator as well

Should it be added in this PR?

src/libraries/sendtohelixhelp.proj

src/libraries/tests.proj

premun · 2021-09-23T09:52:23Z

@steveisok if it's the same to you, let's start with tvOS rather than iOS. We just have way more devices there.

Or if iOS is more important, I'd prefer to do both to be honest. Let's start getting data and see how the tests go and we can start working on infrastructural issues

In an effort to better utilize CI resources, this change will only run the System.Runtime library test per PR for Android and Wasm configurations. The full libraries and runtime tests will move to only run on the post-PR validation pipeline. Note: iOS/tvOS/MacCatalyst are excluded until dotnet#59503 lands

…th in one test

premun · 2021-10-26T10:27:25Z

Nice! The job was 32 hours of work but done within an hour, this is feasible for a rolling build!

eng/testing/tests.targets

eng/testing/tests.ioslike.targets

src/libraries/sendtohelixhelp.proj

src/libraries/System.Net.NetworkInformation/tests/FunctionalTests/IPGlobalPropertiesTest.cs

mdh1418 · 2021-10-26T15:29:31Z

src/libraries/sendtohelixhelp.proj

@@ -87,6 +91,12 @@
    <HelixPostCommand Include="taskkill.exe /f /im corerun.exe"/>
  </ItemGroup>

+  <ItemGroup Condition="'$(TargetOS)' == 'iOS' or '$(TargetOS)' == 'iOSSimulator'">


Should it be added in this PR?

src/libraries/sendtohelixhelp.proj

mdh1418 · 2021-10-26T15:32:43Z

src/mono/msbuild/apple/build/AppleApp.InTree.targets

+  <UsingTask TaskName="MonoAOTCompiler" AssemblyFile="$(MonoAOTCompilerTasksAssemblyPath)" />
+  <UsingTask TaskName="RuntimeConfigParserTask" AssemblyFile="$(MonoTargetsTasksAssemblyPath)" />
+
+  <!-- TODO: this breaks runtime tests on Helix due to the file not being there for some reason. Once this is fixed we can remove the UpdateRuntimePack target here -->


Should we get an issue to track this?

Good question. There may be already. Will look

we should file an issue for this if it doesn't exist.

steveisok · 2021-10-28T12:24:38Z

@akoeplinger Please give this a review when you have a moment.

…60727) In an effort to better utilize CI resources, this change will only run the System.Runtime library test per PR for Android and Wasm configurations. The full libraries and runtime tests will move to only run on the post-PR validation pipeline. The summary of changes are: - System.Runtime tests will only trigger per PR for Android x86/x64 and Wasm AOT / EAT - All mono runtime builds will only run on the post PR validation build (rolling build) - WasmBuildTests will only run on the post PR validation build (rolling build) - Wasm on Windows will only build Note: iOS/tvOS/MacCatalyst are excluded until #59503 lands

akoeplinger

Left a few comments but looks great overall :)

eng/pipelines/runtime-staging.yml

akoeplinger · 2021-10-29T20:49:16Z

eng/testing/tests.mobile.targets

      <ResolvedFileToPublish TrimMode="$(TrimMode)" />

      <!-- Don't trim the main assembly.
           TrimMode="" is needed so the root assemblies are correctly identified -->
      <ResolvedFileToPublish TrimMode="" Condition="'%(FileName)' == '$(AssemblyName)'" />

      <!-- TODO: find out why these assemblies aren't copied by copyused even though they're referenced -->
+      <ResolvedFileToPublish TrimMode="Copy" Condition="'$(EnableSoftTrimming)' == 'true' and '%(FileName)' == 'System.ComponentModel.EventBasedAsync'" />


looks like this is still unaddressed, we should have a tracking issue for it

src/libraries/Common/tests/TestUtilities/System/PlatformDetection.cs

src/libraries/Common/tests/Extensions/ConfigurationRootTest.cs

src/libraries/System.ComponentModel.TypeConverter/tests/CustomTypeDescriptorTests.cs

src/libraries/System.ComponentModel.TypeConverter/tests/TypeDescriptionProviderTests.cs

src/libraries/System.Net.NetworkInformation/tests/FunctionalTests/IPGlobalPropertiesTest.cs

src/libraries/System.Private.Xml/tests/Xslt/XslTransformApi/XSLTransform.cs

src/libraries/System.Runtime.Loader/tests/System.Runtime.Loader.Tests.csproj

akoeplinger · 2021-10-29T21:16:55Z

src/mono/msbuild/apple/build/AppleApp.InTree.targets

+  <UsingTask TaskName="MonoAOTCompiler" AssemblyFile="$(MonoAOTCompilerTasksAssemblyPath)" />
+  <UsingTask TaskName="RuntimeConfigParserTask" AssemblyFile="$(MonoTargetsTasksAssemblyPath)" />
+
+  <!-- TODO: this breaks runtime tests on Helix due to the file not being there for some reason. Once this is fixed we can remove the UpdateRuntimePack target here -->


we should file an issue for this if it doesn't exist.

steveisok · 2021-11-02T09:31:38Z

I'll add the issues and reference them in a separate PR.

Steve Pfister added 12 commits August 12, 2021 23:33

iOS AOT run w/ smaller test files

64ef141

See if zipping each test app first results in not running out of disk space. Changed cleanup steps after each test is built to maximize space utilization.

Merge remote-tracking branch 'upstream/main' into test-zip-app

f7f87f7

Bump timeout. Skip some tests that fail aot

2e1ee0e

Merge remote-tracking branch 'upstream/main' into test-zip-app

336ec94

Print out space used and number of tests processed

9b29dc5

Structure iOS test build to be able to run on helix

4d678d5

Get past race by enabling logging. Test passes on helix now

c42ef12

Enable building on helix for other apple configurations

c69a4cd

Integrate soft trimming

5324672

Update template to work with arcade changes

f052234

Merge remote-tracking branch 'upstream/main' into build-ios-tests-helix

fe39010

Post merge with main adjustments

d610108

dotnet-issue-labeler bot added the area-Infrastructure-libraries label Sep 22, 2021

steveisok requested review from akoeplinger, MaximLipnin, directhex, mdh1418 and radical September 22, 2021 21:36

steveisok force-pushed the build-ios-tests-helix branch from 785bb9c to f7bb825 Compare September 22, 2021 23:20

MaximLipnin reviewed Sep 23, 2021

View reviewed changes

Steve Pfister added 2 commits September 23, 2021 16:18

Bump arcade / cleanup

f084ae9

fix sim runs

17a2bb2

runfoapp bot mentioned this pull request Sep 24, 2021

emsdk failure: emcmake not found #59556

Closed

MaximLipnin mentioned this pull request Sep 27, 2021

Add the argument checks to Unix-part of Mutex.OpenExistingWorker #59634

Merged

Build tweaks to fix running tests locally

e5d84bc

Skip test suites that hang for now

2419858

steveisok marked this pull request as ready for review October 21, 2021 12:51

steveisok requested a review from marek-safar as a code owner October 21, 2021 12:51

Steve Pfister added 2 commits October 21, 2021 09:13

Restored fact that was mistakenly removed

055b85d

Feedback

81bd7fd

steveisok mentioned this pull request Oct 21, 2021

Alter amount of tests that run per PR for mono mobile configurations #60727

Merged

Steve Pfister added 6 commits October 21, 2021 22:17

Skip DllImportGenerator tests as they were failing

1a3b910

Search top level only for zip files

3c1babf

Skip DllImportGenerator tests on all apple mobile platforms. Tweak pa…

fdb81ff

…th in one test

Merge remote-tracking branch 'upstream/main' into build-ios-tests-helix

7d43598

Fix check specific platform check for catalyst

e7d92da

Remove old condition that was breaking unrelated projects

633dcf7

mdh1418 reviewed Oct 26, 2021

View reviewed changes

Steve Pfister added 3 commits October 26, 2021 12:53

Feedback

ef910cf

Merge remote-tracking branch 'upstream/main' into build-ios-tests-helix

4800f60

Fix process test introduced in dotnet#59562

df80bf0

steveisok mentioned this pull request Oct 29, 2021

Mono CI Utilization Improvements #61017

Closed

6 tasks

akoeplinger approved these changes Oct 29, 2021

View reviewed changes

PR Feedback

0fb5d0b

Steve Pfister added 2 commits November 2, 2021 05:36

Merge remote-tracking branch 'upstream/main' into build-ios-tests-helix

6c3c9ee

Apply smoke test setting to iOS & tvOS devices

1b75f13

steveisok merged commit 8c6aeea into dotnet:main Nov 2, 2021

ghost locked as resolved and limited conversation to collaborators Dec 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bring up full library test CI runs on iOS / tvOS devices #59503

Bring up full library test CI runs on iOS / tvOS devices #59503

steveisok commented Sep 22, 2021

ghost commented Sep 22, 2021

steveisok commented Sep 22, 2021

MaximLipnin Sep 23, 2021

steveisok Oct 21, 2021

mdh1418 Oct 26, 2021

premun commented Sep 23, 2021 •

edited

Loading

premun commented Oct 26, 2021

mdh1418 Oct 26, 2021

mdh1418 Oct 26, 2021

steveisok Oct 26, 2021

akoeplinger Oct 29, 2021

steveisok commented Oct 28, 2021

akoeplinger left a comment

akoeplinger Oct 29, 2021

akoeplinger Oct 29, 2021

steveisok commented Nov 2, 2021

Bring up full library test CI runs on iOS / tvOS devices #59503

Bring up full library test CI runs on iOS / tvOS devices #59503

Conversation

steveisok commented Sep 22, 2021

ghost commented Sep 22, 2021

steveisok commented Sep 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

premun commented Sep 23, 2021 • edited Loading

premun commented Oct 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveisok commented Oct 28, 2021

akoeplinger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveisok commented Nov 2, 2021

premun commented Sep 23, 2021 •

edited

Loading