Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] speed up conda setup for macOS and Linux jobs #5743

Merged
merged 3 commits into from
Feb 26, 2023
Merged

Conversation

jameslamb
Copy link
Collaborator

@jameslamb jameslamb commented Feb 25, 2023

Follow-up to #5668.

Proposes combining the conda create and conda install steps run in CI jobs into a single conda create step. In #5668 and #5648, we observed that doing that led to less overall time in CI spent waiting for conda solves.

benefits of this change

I observed the following timing differences in these jobs on this PR, compared to master:

job latest master this PR
(GHA) CUDA 11.7 wheel 12m41s 12m16s
(GHA) CUDA 10.0 source 11m17s 11m17s
(GHA) CUDA 11.7.1 pip 12m29s 12m35s
(GHA) macOS regular 16m46s 14m49s
(GHA) macOS bdist 18m33s 13m07s
(GHA) macOS sdist 13m16s 10m06s
(GHA) MPI source 16m52s 12m33s
(GHA) MPI pip 12m21s 10m29s
(GHA) MPI wheel 13m42s 13m03s
(Azure) Linux regular 14m58s 15m21s
(Azure) Linux sdist 13m29s 12m49s
(Azure) Linux bdist 16m08s 15m37s
(Azure) Linux mpi_source 10m47s 10m45s
(Azure) Linux gpu_source 16m13s 17m21s
(Azure) Linux_latest regular 16m20s 15m15s
(Azure) Linux_latest sdist 13m44s 13m52s
(Azure) Linux_latest bdist 16m38s 15m31s
(Azure) Linux_latest mpi_source 09m55s 09m53s
(Azure) Linux_latest gpu_source 16m16s 15m21s
(Azure) Linux_latest gpu_pip 16m52s 15m10s
(Azure) Linux_latest gpu_wheel 16m40s 15m41s
(Azure) QEMU_multiarch bdist 71m29s 70m41s
(Azure) macOS regular 12m08s 11m24s
(Azure) macOS sdist 10m06s 08m31s
(Azure) macOS bdist 11m12s 08m46s

this PR:

latest master build:

Notes for Reviewers

Given all the things that can change from run-to-run, it's hard to trust differences in the timings based on 1 or even a few of runs. But I'm still pretty confident that these changes make these CI jobs at least a small amount faster, without reducing their stability.

@jameslamb jameslamb changed the title WIP: [ci] speed up conda setup for macOS and Linux jobs [ci] speed up conda setup for macOS and Linux jobs Feb 26, 2023
@jameslamb jameslamb marked this pull request as ready for review February 26, 2023 03:58
@jameslamb jameslamb merged commit 0007343 into master Feb 26, 2023
@jameslamb jameslamb deleted the ci/faster-conda branch February 26, 2023 17:07
@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants