Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reducing CI runs #14553

Merged
merged 5 commits into from
Jan 10, 2023
Merged

Reducing CI runs #14553

merged 5 commits into from
Jan 10, 2023

Conversation

T-Gro
Copy link
Member

@T-Gro T-Gro commented Jan 6, 2023

  1. Compressing metadata is stable. Reduce the matrix for test runs to ONLY use the compressed version, and do not do the same on the regular one
  2. The same can be applied for trimming, again only trim the compressed metadata version
  3. FSharpSuite tests are the dominating part of the longest CI run. Since the total wait time scales with the longest job, it makes sense to optimize here
    • This suite will be invocated separately as a background process, while other tests run

FIY, this is timing of the different test suites used in the longest CI job:

Suite Timing
Fsharp.Core.UnitTests Time Elapsed 00:03:48.30
FsharpSuite.Tests Time Elapsed 00:52:02.98
Fsharp.Compiler.ComponentTests Time Elapsed 00:07:31.74
Fsharp.Compiler.UnitTests Time Elapsed 00:00:19.16
Fsharp.Compiler.Service.Tests Time Elapsed 00:01:11.56
Fsharp.Compiler.Private.Scripting.UnitTests Time Elapsed 00:01:20.15
Fsharp.Build.UnitTests Time Elapsed 00:00:04.33

Proof for starting FsharpSuite.Tests on the background:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=129075&view=logs&j=09cc464c-89cd-5758-006d-ba9e3958cfa9&t=098678ff-b2f3-5c88-44d3-4c7be2f47e75&l=2813

(FsharpSuite.Tests is started as a spinoff, then all the smaller/faster test suites happen sequentially, then it waits for FsharpSuite.Tests to finish)

@vzarytovskii
Copy link
Member

I'd leave trimming tests intact for now. We're mostly testing that trimmer/linker can process both and not regressed.

@T-Gro
Copy link
Member Author

T-Gro commented Jan 6, 2023

I'd leave trimming tests intact for now. We're mostly testing that trimmer/linker can process both and not regressed.

It's still running, just only with the compressed version.
The compression is about binary resources...

@vzarytovskii
Copy link
Member

vzarytovskii commented Jan 6, 2023

I'd leave trimming tests intact for now. We're mostly testing that trimmer/linker can process both and not regressed.

It's still running, just only with the compressed version.
The compression is about binary resources...

No, what I meant is that we should leave both of them running.

We want to make sure uncompressed metadata is trimmed correctly too, since it's to stay in some libraries for a long time.

For example, FSharp.Core shipped uncompressed, and will be uncompressed for some time.

azure-pipelines.yml Outdated Show resolved Hide resolved
@KevinRansom
Copy link
Member

KevinRansom commented Jan 6, 2023

Compilers supporting compression have been available since early november. dotnet sdk 7.0.0. VS 17.4.0.
I think I want to turn it on compression by default for FSharp.Core in VS 17.6.0. In the mean time, I'm okay with running the tests with compressed FSharp.Core and Uncompressed FSharp.Core. If we are getting pressed because ci takes too many resources, then we can remove the tests with Compressed FSharp.Core rather than uncompressed, because Compressed FSharp.Core isn't a customer scenario yet.

@T-Gro
Copy link
Member Author

T-Gro commented Jan 9, 2023

Compilers supporting compression have been available since early november. dotnet sdk 7.0.0. VS 17.4.0. I think I want to turn it on compression by default for FSharp.Core in VS 17.6.0. In the mean time, I'm okay with running the tests with compressed FSharp.Core and Uncompressed FSharp.Core. If we are getting pressed because ci takes too many resources, then we can remove the tests with Compressed FSharp.Core rather than uncompressed, because Compressed FSharp.Core isn't a customer scenario yet.

Ok, trimming test stays in doubled mode then, change already done.

@T-Gro T-Gro marked this pull request as ready for review January 9, 2023 08:34
@T-Gro T-Gro requested a review from a team as a code owner January 9, 2023 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants