Pinning python version in the created environment

[jaimergp]

Just ran into a weird issue. Some complex environment files will happily downgrade (and I guess also upgrade) Python versions to solve the environment.

% conda create -n py39 python=3.9
% conda activate py39
% conda env update -f teachopencadd.yml

, where teachopencadd.yml is this file.

That example above will downgrade Python to 3.8, silently! As a result, my CI branch is NOT testing Python 3.9 at all! See this in live action here.

To avoid that problem (there's no conda-env flag to avoid that behavior), I suggest pinning Python in that environment with

echo "python 3.9.*" > $CONDA_PREFIX/conda-meta/pinned

Even with that pinning, conda env STILL ignores that but at least a warning is now shown:

 - WARNING conda.core.solve:_add_specs(611): pinned spec python=3.9 conflicts with explicit specs.  Overriding pinned spec.

We might want to catch that warning and thrown an error instead because this kind of behavior in a CI context is very unexpected.

Alternatively, we might want to check the output of python -V once everything is installed and see if it matches python-version, but I think that's a bit more complicated?

Question is: should we always pin Python if python-version was specified? I can't think of any example where not doing this would lead to surprising behaviour, but happy to discuss!

[jaimergp]

I added an extra CI step for now:

      #  After setup-miniconda
      - name: Ensure we have the right Python
        shell: bash -l {0}
        run: |
          echo "Installed Python: $(python --version)"
          echo "Expected: ${{ matrix.python-version }}"
          python --version | grep "Python ${{ matrix.python-version }}"

• Result

[goanpeca]

Hi @jaimergp, thanks for the thorough report and explanation. This is indeed an issue I was worrying some day will happen :), having the loose python version is not a guarantee (in very specific environments)

We could look at and test

https://docs.conda.io/projects/conda/en/latest/configuration.html

# # pinned_packages (sequence: primitive)
# #   env var string delimiter: '&'
# #   A list of package specs to pin for every environment resolution. This
# #   parameter is in BETA, and its behavior may change in a future release.
# # 
# pinned_packages: []

We could also inject the dependency on the yaml file. If no yaml file is used, well then it means the user is in charge of intaling hs things.

Even with that pinning, conda env STILL ignores that but at least a warning is now shown:

Yeah that is a very odd behavior... and still there... conda/conda#9016

A pin is a pin, and specified in a file there is no reason for it to be ignored., unless --no-pin was used.

Question is: should we always pin Python if python-version was specified? I can't think of any example where not doing this would lead to surprising behaviour, but happy to discuss!

Can't think of one either, so yes either pin + updating the yaml file by injecting the python=python-version. I think the yaml one would be much easier and actually failed with the right error if not possible to satisfy.

[bollwyvl]

If your yaml explicitly doesn't agree with the python-version, yeah it'll be hard. Updating is reasonable, though as #107 already does this to avoid the extra solve. Quick fix against v2 might be to add a condarc and set a few things, but at least: freezeInstalled: true This is probably more drastic than needed, and may have other side effects. But if it works, we could raise it to an action input. In the future, being able to set ALL of condarc YAML directly in "with" might be useful, and we could get rid of a lot of one-off inputs.

[goanpeca]

Hi @bollwyvl thanks for the input.

I prefer exposing one off options than the yaml in yaml option.

These options are fine as they are conda config options so except for the "complex" ones the rest should probably be exposed to be used in the with:

Unless I did not understand things correctly 🙃

[bollwyvl]

My point is: there are over a hundred condarc options, some of which are complex types (e.g objects, ordered lists). actions.yml is a pretty poor specification language, even weaker than JSON Schema, and we do very little checking on them _anyhow_ because everything comes in as "a string". So, as a user, to figure out what to type, first you have to check our docs (maybe) for the name-mangled version, and maybe if we do, you have to use commas to separate things. if we don't support it, then maybe go read the conda docs for _another_ name, and the actual documentation of the value, write a file, set _another_ key, etc. Just saying, "go read the conda docs" and "write YAML like you find there" might be easier and more robust over time. We even already have a YAML parser sitting around! Further, skipping the conda cli, and just writing the condarc directly, will avoid weirdness like doing the channels in reverse order, and allow for the complex keys. But of course, on v2, we can't deprecate our special snowflakes... v3 could add (one) warning, and v4 could purge them... But we might just be stuck with them.

[goanpeca]

Ok I understood correctly :-p

actions.yml is a pretty poor
specification language, even weaker than JSON Schema, and we do very little
checking on them anyhow because everything comes in as "a string".

True, may be extra work on our side to maintain but providing options, keeps providing the best UX and that is important too.

[bollwyvl]

Whew, "best UX" would likely be a whole shooting match of gathering a bunch of observational data (e.g. survey, poll, etc).

[bollwyvl]

As of master, and at some complexity, if python-version is specified along with a YAML file, it will actually build a temporary file with the full spec to python specified:

https://github.com/conda-incubator/setup-miniconda/blob/master/src/env/yaml.ts#L44

This should save us both the initial env spin-up, as well as only consider your new, confected environment, in this case with - python=3.9.

We should probably add a test for this scenario.

[goanpeca]

Just checking @bollwyvl, is this solved by the latest work you did?

[bollwyvl]

Yeah, the one-before-last-of-107 does it with the patched yaml, so just one solve. The test may not be complete, we should probably add an env.yml with a python, and verify it overloads it.

[huangliang0828]

I meet the same problems in win10 conda64 python3.8 & python 3.9，the pinned file in conda-meta folder was ignoreed or overrided， I hope there will be a good solution