Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split pipeline build job into x86 and x64 #1852

Merged
merged 19 commits into from
Jan 20, 2022

Conversation

florelis
Copy link
Member

@florelis florelis commented Jan 13, 2022

Change:

This changes the CI build pipeline to build/test x86 and x64 in two separate parallel jobs.

Context:

On #1813 the build failed because the build machine was running out of space. We are using Microsoft Hosted agents for running the pipelines, which have only 10GB free space (according to docs; in practice it seems to be around 12GB). Upon investigation, building the solution was already taking about 9GB since we are building both x86 and x64 at the same time. Out of these, about 4GB were from pre-compiled headers; the biggest offender was the one for CommonCore, with over 800MB for each x86,x64.

I also considered trying to reduce the size of the pre-compiled headers or delete them after compilation (but didn't know how); or setting up a custom agent pool with more space (but setting it up and maintaining it seemed complicated). The idea of splitting the jobs by platform is from @yao-msft :)

By splitting into two jobs, building the solution for each one now takes only about 5GB, which puts us a safe distance from the 10GB limit.

Change details:

  • Added a matrix strategy for the build job to run the same steps for both platforms
  • Renamed all build artifacts that contained the platform in the file name to now be grouped under a single folder per platform
  • Removed duplicated test runs (there was one per platform)
  • Moved some files generated by the pipeline from the MSBuild output directory to the pipeline's artifact staging directory
  • Changed tests to run only if the build succeeded by removing the succeededOrFailed() condition. Added continueOnError: true to the tests so that it still makes it to the publish test results step if they fail.
  • Also added publishing of MSBuild binlogs for diagnosing build errors

Opening as draft first to validate that the pipeline works correctly.

Microsoft Reviewers: Open in CodeFlow

@github-actions

This comment has been minimized.

@florelis florelis marked this pull request as ready for review January 14, 2022 21:57
@florelis florelis requested a review from a team as a code owner January 14, 2022 21:57
azure-pipelines.yml Outdated Show resolved Hide resolved
@florelis florelis merged commit 8650938 into microsoft:master Jan 20, 2022
@florelis florelis deleted the pipelineJobs branch May 28, 2022 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants