[ci] Appveyor conda env solves are very slow

[jameslamb]

Description

Over the last few weeks, I've observed that each CI job running on Appveyor takes around 25-30 minutes to complete.
That puts a very lower upper limit on how much activity this repo can tolerate, since:

• only one job can be running at a time on our Appveyor account across all commits
• each commit currently triggers two jobs (appveyor.yml link)

Reproducible example

Look at any recent build, for example https://ci.appveyor.com/project/guolinke/lightgbm/builds/45786272.

I downloaded the logs from some builds, and found that a significant amount of time (on the order of 15 minutes) is spent just solving the conda environment in this step:

LightGBM/.ci/test_windows.ps1

Line 54 in fffd066

conda install -q -y -n $env:CONDA_ENV cloudpickle joblib matplotlib numpy pandas psutil pytest "python=$env:PYTHON_VERSION[build=*cpython]" python-graphviz scikit-learn scipy ; Check-Output $?

Look at this trace from that recent build's logs, note the timestamps.

[00:01:29] Collecting package metadata (current_repodata.json): ...working... done
[00:02:21] Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
[00:03:37] Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
[00:05:21] Collecting package metadata (repodata.json): ...working... done
[00:13:03] Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
[00:17:04] Solving environment: ...working... done
[00:17:05] 
[00:17:05] ## Package Plan ##
[00:17:05] 
[00:17:05]   environment location: C:\Miniconda3-x64\envs\test-env
[00:17:05] 
[00:17:05]   added / updated specs:
[00:17:05]     - cloudpickle
[00:17:05]     - joblib
[00:17:05]     - matplotlib
[00:17:05]     - numpy
[00:17:05]     - pandas
[00:17:05]     - psutil
[00:17:05]     - pytest
[00:17:05]     - python-graphviz
[00:17:05]     - python=3.7[build=*cpython]
[00:17:05]     - scikit-learn
[00:17:05]     - scipy
[00:17:05] 
[00:17:05] The following packages will be downloaded:
[00:17:05] 
[00:17:05]     package                    |            build
[00:17:05]     ---------------------------|-----------------
[00:17:05]     attrs-22.2.0               |     pyh71513ae_0          53 KB  conda-forge
...
[00:17:05]     zstd-1.5.2                 |       h7755175_4         401 KB  conda-forge
[00:17:05]     ------------------------------------------------------------
[00:17:05]                                            Total:       457.9 MB
[00:17:05] 
[00:17:05] The following NEW packages will be INSTALLED:
[00:17:05] 
[00:17:05]   attrs              conda-forge/noarch::attrs-22.2.0-pyh71513ae_0
...
[00:17:05]   zstd               conda-forge/win-64::zstd-1.5.2-h7755175_4
[00:17:05] 
[00:17:05] The following packages will be SUPERSEDED by a higher-priority channel:
[00:17:05] 
[00:17:05]   setuptools         conda-forge/noarch::setuptools-65.6.3~ --> conda-forge/win-64::setuptools-59.8.0-py37h03978a9_1
[00:17:05] 
[00:18:20] 
[00:18:24] Preparing transaction: ...working... done
[00:18:33] Verifying transaction: ...working... done
[00:18:50] Executing transaction: ...working... done
[00:18:50] Retrieving notices: ...working... done

In that particular job, the job took 33 minutes total to run. About 16 minutes of that was spent just in that conda install.

Additional Comments

Some related conversations that might help with this:

• [ci] Appveyor builds failing #5507
• [ci] Python tests failing with error about numpy image #4204 (comment)

Some things that could help and should be tried:

• moving to a newer version of Python, since Python 3.7 was dropped by conda-forge a few months ago
  • https://conda-forge.org/docs/user/announcements.html#dropping-python-3-7
  • Dropping Python 3.7 conda-forge/conda-forge-pinning-feedstock#2623
• putting aggressive floors or ceilings on some of those dependencies being installed on Appveyor, to limit the search space for conda
• modifying the conda channels used on Appveyor to ensure that the test-env environment contains only conda-forge packages
• just using pip install instead

[jakirkham]

Might also be worth looking at the libmamba solver for Conda

[jameslamb]

Thanks @jakirkham ! We'll consider that in the future.

You might find the changes in #5648 (the PR that closed this issue) interesting too. We were able to cut 15+ minutes out of the conda solve time on our Appveyor jobs there.

Basically summarized like this:

# slow
conda create --name ${ENV_NAME} python=${PYTHON_VERSION}
conda install --name ${ENV_NAME} ${SOME_PACKAGES}

# fast
conda create --name ${ENV_NAME} python=${PYTHON_VERSION} ${SOME_PACKAGES}

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.