Setup conda step takes too long

[ma-sadeghi]

Thank you for creating/maintaining this great Action!

It seems that setting up conda alone takes longer than usual. For example, take a look at this log, the "Set up conda" step. It takes 3 and a half minutes setting up conda (excluding the installation of dependencies). Is that expected or am I doing something wrong?

Thank you once again!

[goanpeca]

Hi @ma-sadeghi, thanks for the kind words and for using the action :)

It seems that setting up conda alone takes longer than usual.

Has this changed in any newer version? When did you notice started noticing this?

Just wondering so we can pinpoint to any potential issue.

---

@conda-incubator/setup-miniconda-team have you noticed anything similiar as described?

[hadim]

Same issue for me using mamba (without installing my deps during the action) and with the latest v2 version.

[goanpeca]

@bollwyvl I could use hand debugging this

[hadim]

In case it helps (I can't share the logs):

    steps:
      - name: Checkout the code
        uses: actions/checkout@v2

      - name: Cache conda
        uses: actions/cache@v1
        env:
          CACHE_NUMBER: 2
        with:
          path: ~/conda_pkgs_dir
          key: conda-${{ runner.os }}-${{ env.CACHE_NUMBER }}-${{ hashFiles('env.yml') }}

      - name: Setup conda
        uses: conda-incubator/setup-miniconda@v2
        with:
          mamba-version: "*"
          channels: conda-forge,defaults
          channel-priority: true
          use-only-tar-bz2: true

[goanpeca]

Thanks @hadim for the example. Will add a PR to check the changes that introduced the regression.

[ma-sadeghi]

Hi @ma-sadeghi, thanks for the kind words and for using the action :)

It seems that setting up conda alone takes longer than usual.

Has this changed in any newer version? When did you notice started noticing this?

Just wondering so we can pinpoint to any potential issue.

@conda-incubator/setup-miniconda-team have you noticed anything similiar as described?

I'm almost sure that it used to be much quicker the first time I used this Action, off the top of my head, maybe it took 1 minute or so setting up conda, now it's at 3:30 minutes. Our project is public, and it's been a while since we're using this Action, so feel free to check out our logs in case that helps debugging the issue: https://github.com/PMEAL/OpenPNM/actions

[bollwyvl]

From the linked logs:

Tue, 10 Nov 2020 21:09:23 GMT /usr/share/miniconda/condabin/conda update conda
Tue, 10 Nov 2020 21:09:54 GMT Collecting package metadata (repodata.json): ...working... done
Tue, 10 Nov 2020 21:10:07 GMT Solving environment: ...working... done
Tue, 10 Nov 2020 21:11:56 GMT
Tue, 10 Nov 2020 21:11:56 GMT ## Package Plan ##

So we have, somewhat unscientifically:

• 26 seconds of getting repodata
• 135 seconds of doing the solve to update conda

The only thing I can think of is conda-forge being added, which puts the solver in over-drive with an extra 10k+ packages (and all their versions).

Some options, off the top of my head:

• in your repo...
  • set auto-update-conda to false
    • you'll kinda get what you get, but you'll usually be fine
  • generate/check in explicit specification with conda-lock
  • use a custom installer
    • yeah, the download might take longer, but the solver won't get invoked
    • you can/should cache this as well, but sourcing one from github will be faster than going out to the internet
• on this repo..
  • add a new action option, which will only pull from a limited selection of channels, e.g. conda-update-channels: defaults
  • add some ::group::Boring stuff ::endgroup:: log output so that top-level lifecycle of the commands this action runs are easier to track (i had to dust off my pandas to do some of the timing stuff)
    • haven't tried in anger... it might
  • offer a micromaba setup (again, saving a couple solves) Micromamba based setup? #75 (comment)

[hadim]

For what it's worth most of the time is spent installing mamba:

Associated config:

      - name: Cache conda
        uses: actions/cache@v2
        env:
          CACHE_NUMBER: 3
        with:
          path: ~/conda_pkgs_dir
          key: conda-${{ runner.os }}-${{ env.CACHE_NUMBER }}-${{ hashFiles('env.yml') }}

      - name: Setup conda
        uses: conda-incubator/setup-miniconda@v2
        with:
          python-version: 3.7
          mamba-version: "*"
          channels: conda-forge,defaults
          channel-priority: true
          use-only-tar-bz2: true
          activate-environment: nova

Is that possible to use something like micromamba instead of miniconda or miniforge?

[hadim]

Looking at the detail the solving step is what takes 99% of the time.

[jaimergp]

If this issue gets addressed, the miniforge route will provide this by default :D

[hadim]

I confirm the recently released MambaForge installer makes the CI faster:

- name: Setup conda
  uses: conda-incubator/setup-miniconda@v2
  with:
    installer-url: https://github.com/conda-forge/miniforge/releases/download/4.9.2-2/Mambaforge-Linux-x86_64.sh
    channel-priority: true
    use-only-tar-bz2: true
    activate-environment: my-env

- name: Anaconda Login
  run: |
    # Bypass the installation of anaconda-client
    TOKEN_DIR="$HOME/.config/binstar"
    TOKEN_PATH="$TOKEN_DIR/https%3A%2F%2Fapi.anaconda.org.token"
    mkdir -p $TOKEN_DIR
    echo -e "${ANACONDA_USER_TOKEN}\c" > $TOKEN_PATH

- name: Install Dependencies
  run: mamba env update -f env.yml

(the snippet also includes a way to login to anaconda without the need to install anaconda-client. Could be helpful for some here.)

[hadim]

See https://github.com/conda-forge/miniforge#mambaforge

[jaimergp]

How does it play with mamba-version? Is it needed? Does it need a check whether an existing mamba is already in place?

[hadim]

Good question and I actually don't know. The new installer ships mamba by default and I use directly without mamba-version.

I need to first login to anaconda first (see snippet) so I don't rely on setup-conda to install my env.

I guess if you want to use setup-conda to install an env file, setup-conda should be modified to indeed check for mamba and bypass its installation if it exists.

Maybe someone from @conda-incubator/setup-miniconda-team can confirm or not?

[bollwyvl]

There are a lot more moving pieces coming up now than just use what's on disk vs get an installer from a URL with a scheme we've been able to predict for half a decade, and the current code's behavior w/r/t to mamba is hard to reason about.

I'll wager, if you provide a mamba-version, it's still going to try to install mamba from conda-forge, even if it's already there, and still use conda for everything except the environment solve (e.g. conda config). These would defeat most gains of already having it on-disk, post-installer. Since such an installer with mamba didn't exist until 13 hours ago, this wasn't a problem... my thinking is we'll likely need something like conda-exe: mamba if you know it's already installed, as even if it is available, you might not want to use it, and tomorrow there may well be The Next Tool (lambada, tarantala) from The Next Installer.

Anyhow, I felt fixing some of these issues requires some pretty drastic restructuring... on #107 (which is becoming #108, #109, ... pending review) I made a best effort attempt at getting us set up for The Next X, where X can be a one/two file PR. Hopefully we can get to where we can start adding features and addressing performance, but I fear not addressing the current technical debt first is going to make the situation worse.

[bollwyvl]

conda-incubator/setup-miniconda@master has now (for a while) supported the combination of:

miniforge-variable: Mambaforge
use-mamba: true

Which should mostly answer the mail here... to the extent that a full conda+python+mamba install can. Micromamba has had a lot of updates recently, but I still haven't been able to evaluate it as to whether we want to support/test it to the same level as the other providers, yet...