Use bazel's downloader for downloading pip packages #1357
Assignees
Labels
Comments
github-merge-queue bot
pushed a commit
that referenced
this issue
Oct 17, 2023
…parate executions (#1487) Before the PR the downloading/building of the wheel and the extraction would be done as a single step, which meant that for patching of the wheel to happen, we would need to do it within the python script. In order to have more flexibility in the approach, this PR splits the process to two separate invocations of the wheel_installer, which incidentally also helps in a case where the downloading of the wheel file can happen separately via http_file. Related issues #1076, #1357
|
This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. |
github-actions
bot
added
the
Can Close?
aignas
added
type: feature request
type: pip
and removed
Can Close?
michaelboulton
added a commit
to michaelboulton/rules_python
that referenced
this issue
Mar 3, 2024
If we can determine that there's a platform agnostic wheel (dep-ver-py3-none-any.whl) then download it using rctx.download, allowing it to be cached. This partially solves bazelbuild#1357 but it still doesn't handle dependencies that have platform specific wheels like https://pypi.org/project/psycopg2/.
michaelboulton
added a commit
to michaelboulton/rules_python
that referenced
this issue
Mar 3, 2024
If we can determine that there's a platform agnostic wheel (dep-ver-py3-none-any.whl) then download it using rctx.download, allowing it to be cached. This partially solves bazelbuild#1357 but it still doesn't handle dependencies that have platform specific wheels like https://pypi.org/project/psycopg2/.
michaelboulton
added a commit
to michaelboulton/rules_python
that referenced
this issue
Mar 3, 2024
If we can determine that there's a platform agnostic wheel (dep-ver-py3-none-any.whl) then download it using rctx.download, allowing it to be cached. This partially solves bazelbuild#1357 but it still doesn't handle dependencies that have platform specific binary wheels like https://pypi.org/project/psycopg2/
michaelboulton
added a commit
to michaelboulton/rules_python
that referenced
this issue
Mar 3, 2024
If we can determine that there's a platform agnostic wheel (dep-ver-py3-none-any.whl) then download it using rctx.download, allowing it to be cached. This partially solves bazelbuild#1357 but it still doesn't handle dependencies that have platform specific binary wheels like https://pypi.org/project/psycopg2/
michaelboulton
added a commit
to michaelboulton/rules_python
that referenced
this issue
Mar 3, 2024
If we can determine that there's a platform agnostic wheel (dep-ver-py3-none-any.whl) then download it using rctx.download, allowing it to be cached. This partially solves bazelbuild#1357 but it still doesn't handle dependencies that have platform specific binary wheels like https://pypi.org/project/psycopg2-binary/
2 tasks
github-merge-queue bot
pushed a commit
that referenced
this issue
Apr 5, 2024
This introduces 3 attributes and the minimal code to be able to download wheels using the bazel downloader for the host platform. This is not yet adding support for targeting a different platform but just allows us to get the wheels for the host platform instead of using `pip`. All of this is achieved by calling the PyPI's SimpleAPI (Artifactory should work as well) and getting the all URLs for packages from there. Then we use the `sha256` information within the requirements files to match the entries found on SimpleAPI and then pass the `url`, `sha256` and the `filename` to `whl_library`, which uses `repository_ctx.download`. If we cannot find any suitable artifact to use, we fallback to legacy `pip` behaviour. Testing notes: * Most of the code has unit tests, but the `pypi_index.bzl` extension could have more. * You can see the lock file for what the output of all of this code would be on your platform. * Thanks to @dougthor42 for testing this using the credentials helper against a private registry that needs authentication to be accessed. Work towards #1357
github-merge-queue bot
pushed a commit
that referenced
this issue
Jun 6, 2024
…epo (#1837) With this change we add support for platform-specific wheel registration and doing the selection of which wheel is used at build time. This supports: - Different package versions for different platforms. - Use string_flags to configure what to fetch/select: - only whls, only sdist or auto mode. - libc version and `musl` vs `glibc` selection. - universal2 vs arch wheels for mac. - target osx version selection. Summary of changes: - The `uv pip install` would only warn the user of yanked packages but would not refuse to install them. Update our implementation to better match the same behaviour. - A previous PR has added the support for passing it in the `requirements_by_platform` and this just add the necessary code to make sure that we can also do the dependency management when parsing the `whl` `METADATA` files. - Only configure `dev_pip` deps for `linux` and `osx` platforms to not raise issues later. - Add a function for generating a `whl_library` name from a `filename`. - Add a macro for generating all config settings for a particular set of parameters. - Update `render_pkg_aliases` to also use those config settings. - Update the toolchain selection `target_settings` to use the `py_linux_libc` config setting. With this the user can register a `musl` linux toolchain if needed. We can also use similar `flag_values` to resolve #1876. - Integrate everything in the `pip` extension and setup cross-platform select statements. Work towards #1357, #260 Stacked on #1937
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🚀 feature request
Relevant Rules
The relevant rule is
whl_library.Description
Currently whl_library downloads pip packages by executing a python script. However this bypasses bazel's downloader, causing us to lose a few nice features such as:
--experimental_downloader_config.Describe the solution you'd like
It would be nice to use bazel's downloader for these, by using
repository_ctx.download.Describe alternatives you've considered
Not doing this and keeping it like it is, I suppose.
The text was updated successfully, but these errors were encountered: