compile_pip_requirements + 3.11 failing on some packages

[aschleck]

🐞 bug report

Affected Rule

The issue is caused by the rule: compile_pip_requirements

Is this a regression?

I don't think this is a regression in rules_python, but this works with Python 3.10 and does not work with 3.11.

Description

When I try to use compile_pip_requirements to recompile a lockfile with certain packages it cannot handle it. I would have thought the libraries were wrong, however a) pip install works in a non-Bazel venv and b) if I manually resolve the lockfile then pip_parse successfully can parse it and the dependencies work in my py_binary targets.

This seems similar-but-different to #914, but enable_implicit_namespace_pkgs does not seem to be an argument to compile_pip_requirements and anyway pip_parse has no problem parsing this package. Plus adding or removing --incompatible_default_to_explicit_init_py has no effect on this problem.

🔬 Minimal Reproduction

WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_python",
    sha256 = "a644da969b6824cc87f8fe7b18101a8a6c57da5db39caa6566ec6109f37d2141",
    strip_prefix = "rules_python-0.20.0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.20.0/rules_python-0.20.0.tar.gz",
)

load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
    name = "python3_11",
    python_version = "3.11",
)

load("@python3_11//:defs.bzl", "interpreter")
load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "python_deps",
    python_interpreter_target = interpreter,
    requirements_lock = "//:requirements_actual.txt",
)

load("@python_deps//:requirements.bzl", install_deps = "install_deps")

install_deps()

BUILD:

load("@rules_python//python:pip.bzl", "compile_pip_requirements")

package(default_visibility = ["//visibility:public"])

compile_pip_requirements(
    name = "requirements_lock",
    extra_args = ["--allow-unsafe"],
)

requirements_lock.in:

docopt==0.6.2

Then run bazel run //:requirements_lock.update

If you create requirements_actual.txt with the same content as requirements_lock.in, then you can bazel build @python_deps_docopt//:pkg or use it as a dependency of a py_binary target succesfully.

🔥 Exception or Error

Updating ./requirements_lock.txt
WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
    error: subprocess-exited-with-error

    × python setup.py egg_info did not run successfully.
    │ exit code: 1
    ╰─> [6 lines of output]
        Traceback (most recent call last):
          File "", line 2, in 
          File "", line 34, in 
          File "/tmp/pip-resolver-fahvht28/docopt/setup.py", line 3, in 
            from docopt import __version__
        ModuleNotFoundError: No module named 'docopt'
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/build/metadata_legacy.py", line 64, in generate_metadata
    call_subprocess(
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
    raise error
pip._internal.exceptions.InstallationSubprocessError: python setup.py egg_info exited with 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/rules_python/python/pip_install/tools/dependency_resolver/dependency_resolver.py", line 143, in 
    cli()
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__click/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/scripts/compile.py", line 580, in cli
    results = resolver.resolve(max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 253, in resolve
    has_changed, best_matches = self._resolve_one_round()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 343, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/resolver.py", line 456, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/local.py", line 87, in get_dependencies
    return self.repository.get_dependencies(ireq)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 238, in get_dependencies
    self._dependencies_cache[ireq] = self.resolve_reqs(
                                     ^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip_tools/piptools/repositories/pypi.py", line 201, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 509, in _resolve_one
    dist = self._get_dist_for(req_to_install)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/resolution/legacy/resolver.py", line 462, in _get_dist_for
    dist = self.preparer.prepare_linked_requirement(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 577, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/distributions/sdist.py", line 61, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/req/req_install.py", line 541, in prepare_metadata
    self.metadata_directory = generate_metadata_legacy(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/april/.cache/bazel/_bazel_april/86326d1c1b8682421b2452cb9add301f/execroot/__main__/bazel-out/k8-fastbuild/bin/requirements_lock.update.runfiles/pypi__pip/pip/_internal/operations/build/metadata_legacy.py", line 71, in generate_metadata
    raise MetadataGenerationFailed(package_details=details) from error
pip._internal.exceptions.MetadataGenerationFailed: metadata generation failed

🌍 Your Environment

Operating System:

  
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"
  

Output of bazel version:

  
Bazelisk version: v1.14.0
Build label: 6.1.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Mar 15 15:44:56 2023 (1678895096)
Build timestamp: 1678895096
Build timestamp as int: 1678895096
  

Rules_python version:

  
0.20.0
  

Anything else relevant?

I'm also seeing this with another package, umi-tools, though there are some other 3.10->3.11 problems there I think so I felt docopt was a cleaner demonstration.

PS thank you so much for maintaining this project! Super amazing to have so much complexity hidden behind relatively simple rules.

[ewianda]

Facing same issue

[xumykargo]

same

[chrislovecnm]

We are upgrading our internal dependencies, so I wonder if it will fix this.

#1322

[qaifshaikh]

Has to do a little with inter-dependencies between packages too. Some very outdated packages seem to cause errors like these and the compile_pip_requirements function can't update packages if they are set to == instead of >=

[brunobely]

Facing the same issue still

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!