"Cross compilation" of py_binary/py_image/py_library targets #260
Comments
|
use https://github.com/ali5h/rules_pip/ and use |
|
@ali5h this works (thanks!) but is there integration into the bazel platform selection functions? I don't want separate targets for Linux and Mac. E.g. Something that works with https://docs.bazel.build/versions/master/platforms.html |
|
you can define multiple piplib repos for different platforms and use select to pickup correct one |
|
Is that documented and supported anywhere? That would be the ideal case, for bazel to automatically pick up the right pip repo for the right target platform. E: should it not just be built in to the py_binary/py_library rule, to select the right target platform libs automatically? |
|
Would really, really like to see this as well. |
|
This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. |
github-actions
bot
added
the
Can Close?
thundergolfer
removed
the
Can Close?
|
Help |
|
Seems #510 and #531 and addressing the same issue? I'd love to have that functionality so excited to so folks interested enough to open PRs 😄 Hopefully @alexeagle and @meastham can combine forces to get the awesome functionality over the line! 🚀 |
|
Friendly ping to @alexeagle and @meastham, are either of you able to take another look at your pull requests? I'd love to have that functionality 😃 |
|
I wonder if pipenv could be used to facilitate generating cross platform dependency graphs. Does anyone have experience with the tool? |
Sorry for the long delay on this! I did some investigation on some of the open questions, I'll see if we can find a path to getting this merged. I don't think the PRs are exactly addressing the same issue. #531 allows having different requirements files for different platforms, but doesn't not allow downloading wheels for a different platform than the host platform. #510 allows downloading wheels for arbitrary platforms ("cross compiling"), but requires every platform to share one requirements file which has potential problems. Conceptually they could be compatible but the implementations would need to be modified to harmonize the way platform selection is done. |
|
I'm all for the wheel-only approach taken in #510. I don't want native compilation happening at all in the loading phase, as it can lead to hard-to-diagnose cache misses between machines (in my experience). Thinking out loud here. If adding a wheel-only option, can we remove I'm sure there are gotchas here (and significant work), but maybe detaching from |
|
I think it should also be possible to compile python/C++ sources into wheels, however those need to happen in actions so they are debuggable and so that the target platform can be used. |
|
Ping on this @alexeagle and @meastham. #510 is exactly what I need today and having #531 would be a strong nice to have. Any progress here? Thanks! |
Hey @gopher-maker, I'm just blocked on getting some guidance from a repository owner on how to resolve the outstanding issues with #510 (not a complaint btw, I'm sure everybody is quite busy!). FWIW we've been using it for a fairly large Python codebase without significant problems for about 9 months now, so if you're feeling adventurous you can use it already. It looks like it now needs some non-trivial rebasing work; I'll see if I can get to that this week. |
|
Any leads on this yet? I am running into a similar issue trying to use Mujoco in a bazel workspace. |
|
@f0rmiga has been working on something related to this at a client, I don't have any update sorry. |
|
There's a current effort that @philsc is writing in a doc, and has the collaboration from @jvolkman. I wrote a resolver to download Python packages using http_file. It's similar to how Gazelle does it for Go third-party deps. I'll start a draft PR in the coming days to have it maintained in rules_python. It will change quite a bit the current workflow to work with wheels in Bazel, and I don't have all the answers yet, so I don't foresee it being in a release soon. |
It seems that a few things broke in recent commits:
- We are not using the `MODULE.bazel.lock` file and it seems that it is
easy to
miss when the components in the PyPI extension stop integrating well
together. This happened during the switch to `{abi}_{os}_{plat}` target
platform passing within the code.
- The logger code stopped working in the extension after the recent
additions
to add the `rule_name`.
- `repo_utils.getenv` was always getting `PATH` env var on bazel `6.x`.
This PR fixes both cases and updates docs to serve as a better reminder.
By
fixing the `select_whls` code and we can just rely on target platform
triples
(ABI, OS, CPU). This gets one step closer to maybe supporting optional
`python_version` which would address #1708. Whilst at it we are also
adding
different status messages for building the wheel from `sdist` vs just
extracting or downloading the wheel.
Tests:
- Added more unit tests and brought them in line with the rest of the
code.
- Checked manually for differences between the `MODULE.bazel.lock` files
in our
`rules_python` extension before #2069 and after this PR and there are no
differences except in the `experimental_target_platforms` attribute in
`whl_library`. Before this PR you would see that we do not select any
wheels
for e.g. `MarkupSafe` and we are always building from `sdist`.
Work towards #260.
|
Before closing the issue we should ensure that we no longer depend on the |
Before this change the `all_requirements` and related constants will have packages that need to be installed only on specific platforms and will mean that users relying on those constants (e.g. `gazelle`) will need to do extra work to exclude platform-specific packages. The package managers that that support outputting such files now include `uv` and `pdm`. This might be also useful in cases where we attempt to handle non-requirements lock files. Note, that the best way to handle this would be to move all of the requirements parsing code to Python, but that may cause regressions as it is a much bigger change. This is only changing the code so that we are doing extra processing for the requirement lines that have env markers. The lines that have no markers will not see any change in the code execution paths and the python interpreter will not be downloaded. We also use the `*_ctx.watch` API where available to correctly re-evaluate the markers if the `packaging` Python sources for this change. Extra changes that are included in this PR: - Extend the `repo_utils` to have a method for `arch` getting from the `ctx`. - Change the `local_runtime_repo` to perform the validation not relying on the implementation detail of the `get_platforms_os_name`. - Add `$(UV)` make variable for the `uv:current_toolchain` so that we can generate the requirements for `sphinx` using `uv`. - Swap the requirement generation using `genrule` and `uv` for `sphinx` and co so that we can test the `requirement` marker code. Note, the `requirement` markers are not working well with the `requirement_cycles`. Fixes #1105. Fixes #1868. Work towards #260, #1975. Related #1663. --------- Co-authored-by: Richard Levasseur <[email protected]>
…when build sdist (#2126) Building sdist results in `Could not find a version that satisfies the requirement setuptool` this regressed when a fix in parameter handling got introduced in #2091. Before this change the building from sdist when using `experimental_index_url` would break because `--no-index` is passed to `pip`. This means that `pip` would fail to locate build time dependencies needed for the packages and would just not work. In `whl_library` we setup `PYTHONPATH` to have some build dependencies available (like `setuptools`) and we could use them during building from `sdist` and to do so we need to add `--no-build-isolation` flag. However, for some cases we need to also add other build-time dependencies (e.g. `flit_core`) so that the building of the wheel in the `repository_rule` context is successfuly. Removing `--no-index` allows `pip` to silently fetch the needed build dependencies from PyPI if they are missing and continue with the build. This is not a perfect solution, but it does unblock users to use the `sdist` distributions with the experimental feature enabled by using `experimental_index_url` (see #260 for tracking of the completion). Fixes #2118 Fixes #2152 --------- Co-authored-by: aignas <[email protected]>
With this change we set the default value of `--python_version` when the `python.toolchain` is used in `bzlmod` and we generate the appropriate config settings based on the registered toolchains and given overrides by the root module. This means that we expect the `--python_version` to be always set to a non-empty value under `bzlmod` and we can cleanup code which was handling `//conditions:default` case. This also means that we can in theory drop the requirement for `python_version` in `pip.parse` and just setup dependencies for all packages that we find in the `requirements.txt` file and move on. This is left as future work by myself or anyone willing to contribute. We can also start reusing the same `whl_library` instance for multi-platform packages because the python version flag is always set - this will simplify the layout and makes the feature non-experimental anymore under bzlmod. Summary: * Add `@pythons_hub` to the `WORKSPACE` with dummy data to make pythons_hub work. * Add `MINOR_MAPPING` and `PYTHON_VERSIONS` to pythons_hub to generate the config settings. * Remove handling of the default version in `pypi` code under bzlmod. Work towards #2081, #260, #1708 --------- Co-authored-by: Richard Levasseur <[email protected]>
With this change we can support ibazel for building our docs again because we will just not have any sdists that are causing issues. This limits the scope to only supporting whls at this time, but for the time being it is the best solution. Fixes bazelbuild#2223 Work towards bazelbuild#260
Before this change the `extra_pip_args` would not be passed down the chain if `experimental_index_url` is set. This adds a test and fixes the bug. Work towards bazelbuild#260.
Before this change the `extra_pip_args` would not be passed down the chain if `experimental_index_url` is set. This adds a test and fixes the bug. Work towards #260
…repos (#2369) Before this change, it was impossible for users to use the targets created with `additive_build_content` whl annotation unless they relied on the implementation detail of the naming of the spoke repositories and had `use_repo` statements in their `MODULE.bazel` files importing the spoke repos. With #2325 in the works, users will have to change their `use_repo` statements, which is going to be disruptive. In order to offer them an alternative for not relying on the names of the spokes, there has to be a way to expose the extra targets created and this PR implements a method. Incidentally, the same would have happened if we wanted to stabilize the #260 work and mark `experimental_index_url` as non-experimental anymore. I was hoping we could autodetect them by parsing the build content ourselves in the `pip` extension, but it turned out to be extremely tricky and I figured that it was better to have an API rather than not have it. Whilst at it, also relax the naming requirements for the `whl_modifications` attribute. Fixes #2187
…azelbuild#2325) With this change we finally generate the same lock file within the legacy code `pip.parse` code path and it allows to slowly transition to using the new code path as much as possible without user doing anything. This moves the selection of the host-compatible lock file from the extension evaluation to the build phase - note, we will generate extra repositories here which might not work on the host platform, however, if the users are consuming the `whl_library` repos through the hub repo only, then everything should work. A known issue is that it may break `bazel query` and in these usecases it is advisable to use `cquery` until we have `sdist` cross-building from source fully working. Summary: - feat: reuse the `render_pkg_aliases` for when filename is not known but platform is known - feat: support generating the extra config settings required - feat: `get_whl_flag_versions` now generates extra args for the rules - feat: make lock file generation the same irrespective of the host platform - test: add an extra test with multiple requirements files - feat: support cross-platform builds using `download_only = True` in legacy setups Note, that users depending on the naming of the whl libraries will need to start using `extra_hub_aliases` attribute instead to keep their setups not relying on this implementation detail. Fixes bazelbuild#2268 Work towards bazelbuild#260 --------- Co-authored-by: Richard Levasseur <[email protected]>
This just cleans up the code and moves more logic from the repository_rule (i.e. generation of `BUILD.bazel` files) to loading time (macro evaluation). This makes the unit testing easier and I plan to also move the code that is generating config setting names from filenames to this new macro, but wanted to submit this PR to reduce the review chunks. Summary: - Add a new `pkg_aliases` macro. - Move logic and tests for creating WORKSPACE aliases. - Move logic and tests bzlmod aliases. - Move logic and tests bzlmod aliases with groups. - Add a test for extra alias creation. - Use `whl_alias` in `pypi` extension integration tests. - Improve the serialization of `whl_alias` for passing to the pypi hub repo. Related to #260, #2386, #2337, #2319 - hopefully cleaning the code up will make it easier to address those feature requests later. --------- Co-authored-by: Richard Levasseur <[email protected]>
Hi,
I have a py_binary that depends on a python pip library (grpcio) that has a native extension bundled in. This means that to create a linux container i'd need to have the pip_import rule download the manylinux wheel, not the host one (macos in my case).
Is there a way to force this? Otherwise py_image will happily just bundle up wheels with darwin native libs. py_binary will also only make host-runnable things.
The text was updated successfully, but these errors were encountered: