Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support patch-level Python versions #1371

Closed
rickeylev opened this issue Aug 10, 2023 · 2 comments · Fixed by #1830
Closed

Support patch-level Python versions #1371

rickeylev opened this issue Aug 10, 2023 · 2 comments · Fixed by #1830

Comments

@rickeylev
Copy link
Collaborator

rickeylev commented Aug 10, 2023

The basic feature request is to allow patch-level specificity of Python versions e.g. python.toolchain(python_version="3.1.2").

Supporting this allows builds to be a bit more reproducible, as it removes the variability of different rules_python versions selecting a different patch level. That said, it's also rather niche. Patch level differences are usually inconsequential.

aignas@ also points out:

different Python toolchain versions come from different indygreg toolchain releases and some of them may have different packaging behaviour, like the last release contains the following changelog: <various updates to pip, musl, openssl, sqlite, libxzma, etc>

Making this work cleanly requires addressing a few things, though:

  • If two modules (or configs) specify different patch levels, what happens?
  • Toolchain repositories are named e.g. python_X_Y; these names aren't supposed to be public, but usage of them is fairly common.
  • Version-pinning rules generate //X.Y:defs.bzl files.
  • PyPI repo names contain the X.Y version, e.g. pip_38_foo.
  • Does the packaging ecosystem support patch-level specificity? e.g. pip resolution, environment markers, whl naming formats, etc.

cc @aignas

Few potential ideas:

  • Allow the root module to specify patch-level version. It has precedence. This allows
    projects to pin to specific versions if they want to. Submodules are not allowed to use patch-level specificity; not sure whether to silently ignore, ignore with warning, or error.
@aignas
Copy link
Collaborator

aignas commented Oct 17, 2023

FYI, this is working:

diff --git a/examples/bzlmod/MODULE.bazel b/examples/bzlmod/MODULE.bazel
index 0d1c7a7..9dbdfd5 100644
--- a/examples/bzlmod/MODULE.bazel
+++ b/examples/bzlmod/MODULE.bazel
@@ -18,7 +18,7 @@ python.toolchain(
     configure_coverage_tool = True,
     # Only set when you have mulitple toolchain versions.
     is_default = True,
-    python_version = "3.9",
+    python_version = "3.9.18",
 )

 # We are also using a second version of Python in this project.
@@ -35,7 +35,7 @@ python.toolchain(
 # See the tests folder for various examples on using multiple Python versions.
 # The names "python_3_9" and "python_3_10" are autmatically created by the repo
 # rules based on the `python_version` arg values.
-use_repo(python, "python_3_10", "python_3_9", "python_versions")
+use_repo(python, "python_3_10", "python_versions", python_3_9 = "python_3_9_18")

 # This extension allows a user to create modifications to how rules_python
 # creates different wheel repositories.  Different attributes allow the user
@@ -89,7 +89,7 @@ use_repo(pip, "whl_mods_hub")
 # the Python interpreter to run to resolve dependencies.
 pip.parse(
     hub_name = "pip",
-    python_version = "3.9",
+    python_version = "3.9.18",
     requirements_lock = "//:requirements_lock_3_9.txt",
     requirements_windows = "//:requirements_windows_3_9.txt",
     # These modifications were created above and we

So it means that #1340 can be closed. The defition of done for this ticket would be to document this behaviour.

@martis42
Copy link
Contributor

@aignas @rickeylev
I disagree with this being a documentation issue. The recommended undocumented workaround does not work for me.
One can tell python.toolchain(...) to use a specific patch version which works just fine as long as one does not use pip.parse (aka a project without external dependencies). But pip.parse does not work with the proposed workaround.

See this minimal example:

MODULE.bazel

bazel_dep(
    name = "rules_python",
    version = "0.27.0",
)

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.10.11",
)

# Uncommenting this has no effect on the error
# use_repo(python, "python_versions", python_3_10 = "python_3_10_11")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pip",
    python_version = "3.10.11",
    requirements_lock = "//:requirements.txt",  # content is irrelvant for reproducing error
)
use_repo(pip, "pip")

BUILD

load("@rules_python//python:defs.bzl", "py_binary")
load("@pip//:requirements.bzl", "requirement")

py_binary(
    name = "foo",
    srcs = ["foo.py"],
    deps = [requirement("some_dep")],  # concrete dep is irrelevant for error
)

Executing bazel run //:foo yields the error:

ERROR: Traceback (most recent call last):
        File ".../external/rules_python~0.27.0/python/private/bzlmod/pip.bzl", line 298, column 30, in _pip_impl
                _create_whl_repos(module_ctx, pip_attr, hub_whl_map, whl_overrides)
        File ".../external/rules_python~0.27.0/python/private/bzlmod/pip.bzl", line 91, column 17, in _create_whl_repos
                fail((
Error in fail: Unable to find interpreter for pip hub 'pip' for python_version=3.10.11: Make sure a corresponding `python.toolchain(python_version="3.10.11")` call exists
ERROR: error evaluating module extension pip in @@rules_python~0.27.0//python/extensions:pip.bzl

I am using rules_python 0.27.0 and Bazel 7.0.0 on Linux Mint 21.1

github-merge-queue bot pushed a commit that referenced this issue Apr 1, 2024
Before only X.Y would work and with this change users can
also specify the exact python version.

Fixes #1371
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants