Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pip.parse parsing all requirements files fails on platform specific wheel downloads #2450

Closed
keith opened this issue Nov 27, 2024 · 5 comments · Fixed by #2488
Closed

pip.parse parsing all requirements files fails on platform specific wheel downloads #2450

keith opened this issue Nov 27, 2024 · 5 comments · Fixed by #2488
Assignees
@aignas

Comments

@keith
Copy link
Member

keith commented Nov 27, 2024

🐞 bug report

Affected Rule

On this branch: #2449 which sets up pypi for 3 different platforms with different requirements files. Running bazel query 'deps(...)' results in a wheel download failure if you're not on x86_64 linux, since the deps for that platform are not available on other platforms:

===== stderr end ===== and referenced by '@@rules_python~~pip~pip//torch:pkg'
ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python~~pip~pip_311_torch_linux_x86_64//':
rules_python:whl_library(@@rules_python~~pip~pip_311_torch_linux_x86_64) FAIL: repo.execute: whl_library.ResolveRequirement(rules_python~~pip~pip_311_torch_linux_x86_64, torch==2.4.1+cpu): end: failure:
  command: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_host/python -m python.private.pypi.whl_installer.wheel_installer --requirement torch==2.4.1+cpu --isolated --extra_pip_args "{\"arg\":[\"--index-url\",\"https://pypi.org/simple\",\"--extra-index-url\",\"https://download.pytorch.org/whl/cpu\"]}" --pip_data_exclude "{\"arg\":[]}" --environment "{\"arg\":{}}"
  return code: 1
  working dir: <default: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~pip~pip_311_torch_linux_x86_64>
  timeout: 600
  environment:
PYTHONPATH="/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__build:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__click:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__colorama:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__importlib_metadata:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__installer:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__more_itertools:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__packaging:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pep517:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pip:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pip_tools:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pyproject_hooks:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__setuptools:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__tomli:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__wheel:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__zipp"
CPPFLAGS="-isystem /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_host/include/python3.11"
===== stdout start =====
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
===== stdout end =====
===== stderr start =====
ERROR: Could not find a version that satisfies the requirement torch==2.4.1+cpu (from versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.5.1)
ERROR: No matching distribution found for torch==2.4.1+cpu
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_aarch64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_host/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--index-url', 'https://pypi.org/simple', '--extra-index-url', 'https://download.pytorch.org/whl/cpu', '-r', '/tmp/tmprruvzh7j']' returned non-zero exit status 1.
===== stderr end =====

Is this a regression?

This was caused by this default flip: daed352

Related to https://github.com/bazelbuild/rules_python/issues/keith

Currently @ HEAD you have to revert the root cause of #2446 to repro this issue
@keith keith mentioned this issue Nov 27, 2024
Open
@aignas
Copy link
Collaborator

aignas commented Nov 28, 2024

This is somewhat of a known thing - bazel query forces all of the repos to be downloaded, not only the ones that are needed for a particular platform. There are multiple ways this can be solved:

  • Users switch to bazel cquery
  • Users start using download_only = True attribute.
  • Users use download_only = True with experimental_index_url.
  • rules_python strips the env marker, in this case platform_machine == foo from the requirement_line that is passed to whl_library. That will make the fetching not fail, but the result that is materialized within the repo rule would be different depending on the platform.

I personally think we as maintainers can only document what users can do and strip the markers from the whl_library because they don't add any value when fetching the wheels.

Would that be acceptable?

@aignas aignas self-assigned this Nov 28, 2024
@keith
Copy link
Member Author

keith commented Dec 2, 2024

I don't know the full implications of suggestion 3 there, but trying that it doesn't seem to work. I still see failures across platforms for the inability to download the platform specific wheels. I repro'd on this branch #2449 by adding:

    download_only = True,
    experimental_index_url = "https://pypi.org/simple",  # use Bazel downloader

to the MODULE.bazel and running bazel query 'deps(...)'.

@aignas
Copy link
Collaborator

aignas commented Dec 5, 2024

For this to work you would need to have lock files with hashes --generate-hashes needs to be passed to uv. #2023 is tracking to support this without the hashes.

@aignas aignas mentioned this issue Dec 9, 2024
Merged
@keith
Copy link
Member Author

keith commented Dec 9, 2024

nice looks like that does fix that example, although it does produce a whole bunch of these warnings:

DEBUG: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=8800deef0026011d502c0c256cc4b67d002347f63c3a38cd8e45f1f445c61364
DEBUG: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=c4f2c3c026e876d4dad7629170ec14fff48c076d6c2ae0e354ab3fdc09024f00

not sure if that's on rules_python or upstream tho

@keith
Copy link
Member Author

keith commented Dec 9, 2024

and then doing bazel test ... on that branch results in:

ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python++pip+pip/torch/BUILD.bazel:6:12: configurable attribute "actual" in @@rules_python++pip+pip//torch:pkg doesn't match this configuration: No matching wheel for current configuration's Python version.

The current build configuration's Python version doesn't match any of the Python
wheels available for this distribution. This distribution supports the following Python
configuration settings:
    //_config:is_cp3.11_cp3x_none_osx_11_0_aarch64
    //_config:is_cp3.11_cp3x_none_osx_aarch64

not sure if that's related

github-merge-queue bot pushed a commit that referenced this issue Dec 12, 2024
@aignas @rickeylev
fix(pypi): pass requirements without env markers to the whl_library (#…
bda710c 
…2488)

With this change the environment markers from the requirements.txt files
no longer end up in the whl_library definitions. I am reusing a function
that already is parsing each requirement line for `sha256` values and
added
logic to extract the `marker` at that point. This means that the change
is
also trivial to backport to the `WORKSPACE` and the logic in the
extension
becomes simpler and we don't rely only on integration tests.

Expected changes to the users:
* If they have vendored pip requirements in `WORKSPACE`, those will be
  reformatted and the env markers will be removed.
* The `MODULE.bazel.lock` file will be likewise reformatted if users are
not using `--experimental_index_url`. Also, the env markers will not be
  passed in the `requirement`.
* `bazel query 'deps("@pypi//foo")'` should start working in more cases.

Fixes #2450.

---------

Co-authored-by: Richard Levasseur <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aignas aignas

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(pypi): pass requirements without env markers to the whl_library aignas/rules_python
2 participants
@aignas @keith