universal locking: allow forking when respecting Python version?

[alex]

~ ❯❯❯ uv --version
uv 0.2.30 (Homebrew 2024-07-26)

Consider this command:

~ ❯❯❯ echo 'nox' | uv pip compile - --universal -p 3.7
warning: The requested Python version 3.7 is not available; 3.12.4 will be used to build dependencies instead.
Resolved 13 packages in 6ms
# This file was autogenerated by uv via the following command:
#    uv pip compile - --universal -p 3.7
argcomplete==3.1.2
    # via nox
colorama==0.4.6 ; sys_platform == 'win32'
    # via colorlog
colorlog==6.8.2
    # via nox
distlib==0.3.8
    # via virtualenv
filelock==3.12.2
    # via virtualenv
importlib-metadata==6.7.0 ; python_version < '3.8'
    # via
    #   argcomplete
    #   nox
    #   virtualenv
nox==2024.4.15
packaging==24.0
    # via nox
platformdirs==4.0.0
    # via virtualenv
tomli==2.0.1 ; python_version < '3.11'
    # via nox
typing-extensions==4.7.1 ; python_version < '3.8'
    # via
    #   importlib-metadata
    #   nox
    #   platformdirs
virtualenv==20.26.3
    # via nox
zipp==3.15.0 ; python_version < '3.8'
    # via importlib-metadata

argcomplete is pinned to version 3.1.2, because this is the newest version that supports Python 3.7. However, there are newer versions (3.4.0) available.

It'd be helpful (though maybe difficult!) if it were possible for --universal to emit "forks" for Python version:

argcomplete==3.1.2; python_version<3.8
argcomplete==3.4.0; python_version>=3.8

[konstin]

This looks similar to #4668.

In general, we try to reduce the number of diverging packages in your resolution (more different behaviors, more versions to audit, large lockfile). If you know that you want to split, you can manually get two distinct resolutions with:

nox; python_version<'3.8'
nox; python_version>='3.8'

Does nox need to be installed into the project venv? If not, would having it as a separate tool work for you? It's mostly like uvx nox is today: Your project and your .venv can use 3.7, while nox gets installed into a separate hidden venv with python 3.12, except that the nox version is specified in pyproject.toml and locked in uv.lock

[alex]

nox was just an example where we see this. That's an interesting note about simply specifying it twice with two different constraints, that makes sense!

[konstin]

Do you have other examples where this happens? Having specific problem cases is very helpful for choosing the right defaults in the resolver.

[alex]

Sure, echo 'mypy' | uv pip compile - --universal -p 3.7 is another example, where typing-extensions is the impacted package.

In general, every dependency in https://github.com/pyca/cryptography/blob/main/ci-constraints-requirements.txt that has a python_version marker is a place where we care about this.

Our overall use case is: We want to pin dependencies in CI for reproducibility. We also want to test against the newest versions (using dependabot). And we want to test on multiple python versions/environments.

Our ideal behavior, therefore, would be a single lockfile that gets us the newest versions for each platform.

[alex]

I should have posted this before, but the trick with specifying the root dep 2x to force forking doesn't seem to work:

~ ❯❯❯ echo "nox; python_version<'3.8'
      nox; python_version>='3.8'" | uv pip compile - --universal -p 3.7
warning: The requested Python version 3.7 is not available; 3.12.5 will be used to build dependencies instead.
Resolved 13 packages in 8ms
# This file was autogenerated by uv via the following command:
#    uv pip compile - --universal -p 3.7
argcomplete==3.1.2
    # via nox
colorama==0.4.6 ; sys_platform == 'win32'
    # via colorlog
colorlog==6.8.2
    # via nox
distlib==0.3.8
    # via virtualenv
filelock==3.12.2
    # via virtualenv
importlib-metadata==6.7.0 ; python_full_version < '3.8'
    # via
    #   argcomplete
    #   nox
    #   virtualenv
nox==2024.4.15
packaging==24.0
    # via nox
platformdirs==4.0.0
    # via virtualenv
tomli==2.0.1 ; python_full_version < '3.11'
    # via nox
typing-extensions==4.7.1 ; python_full_version < '3.8'
    # via
    #   importlib-metadata
    #   nox
    #   platformdirs
virtualenv==20.26.3
    # via nox
zipp==3.15.0 ; python_full_version < '3.8'
    # via importlib-metadata

[charliermarsh]

So, this actually is possible now with a PR I coincidentally merged tonight. It's not yet released though. You can add this to a uv.toml in the working or any parent directory:

environments = ["python_version >= '3.8'", "python_version < '3.8'"]

Or you can put it in a pyproject.toml, like:

[tool.uv]
environments = ["python_version >= '3.8'", "python_version < '3.8'"]

Then uv will solve those two forks, in order. So you end up with:

# This file was autogenerated by uv via the following command:
#    uv pip compile - --universal -p 3.7
argcomplete==3.1.2 ; python_full_version < '3.8'
    # via nox
argcomplete==3.5.0 ; python_full_version >= '3.8'
    # via nox
colorama==0.4.6 ; sys_platform == 'win32'
    # via colorlog
colorlog==6.8.2
    # via nox
distlib==0.3.8
    # via virtualenv
filelock==3.12.2 ; python_full_version < '3.8'
    # via virtualenv
filelock==3.15.4 ; python_full_version >= '3.8'
    # via virtualenv
importlib-metadata==6.7.0 ; python_full_version < '3.8'
    # via
    #   argcomplete
    #   nox
    #   virtualenv
nox==2024.4.15
packaging==24.0 ; python_full_version < '3.8'
    # via nox
packaging==24.1 ; python_full_version >= '3.8'
    # via nox
platformdirs==4.0.0 ; python_full_version < '3.8'
    # via virtualenv
platformdirs==4.2.2 ; python_full_version >= '3.8'
    # via virtualenv
tomli==2.0.1 ; python_full_version < '3.11'
    # via nox
typing-extensions==4.7.1 ; python_full_version < '3.8'
    # via
    #   importlib-metadata
    #   nox
    #   platformdirs
virtualenv==20.26.3
    # via nox
zipp==3.15.0 ; python_full_version < '3.8'
    # via importlib-metadata

[charliermarsh]

(In general, we prioritize solves in an order that attempts to minimize the number of selected versions. We could actually consider adding different strategies for this... I could imagine a strategy where we attempt to select the most recent version for every fork, rather than trying to minimize the number of versions.)

[alex]

Oooh, I'll play with this in the next release!

FWIW, I think our use case is kind of strange in the world of corporate development, but fairly normal in OSS.

[charliermarsh]

This is fixed in the next release without the need for environments.