Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't use .par files for piptool/whltool #184

Closed
brandjon opened this issue Apr 23, 2019 · 2 comments
Closed

Don't use .par files for piptool/whltool #184

brandjon opened this issue Apr 23, 2019 · 2 comments

Comments

@brandjon
Copy link
Member

piptool and whltool are implicit dependencies of the pip_import and whl_library rules, yet they themselves are defined in terms of requirement() declarations. This circular dependency is currently resolved by checking in prebuilt .par files for these tools, regenerated by running update_tools.sh. This has drawbacks:

  1. It makes the build process more brittle and harder to understand.

  2. For security, the checked in par files must only be updated by a trusted person.

See also my recap here, a brittleness breakage here, and an uncompleted PR to migrate away from checked-in .par files here.

@brandjon
Copy link
Member Author

brandjon commented Nov 8, 2019

To clarify, I think there's three separate issues here:

  1. The tools are precompiled binaries, which means they can only be updated by trusted developers.

  2. The tools are built using rules_python itself, so we have to follow a bootstrapping process when updating them.

  3. The tools are par files, which means their build process depends on google/subpar.

For 1, we need these tools to be directly executable because Bazel cannot build its own tools during WORKSPACE evaluation. There's two ways to have an executable Python program: build an artifact (e.g. with Bazel), or else directly execute Python source code without a build step. We currently do the former, but a change like PR #81 does the latter.

For 2, the tools themselves have PyPI dependencies (which happen to be pip, setuptools, and wheel). The current approach of building the tools with rules_python itself provides a convenient way to obtain these dependencies without vendoring, at the cost of added build complexity. #81 switches to vendoring.

For 3, it's not clear to me that the use of par files is necessary, even if we don't vendor. We could explore switching to py_binary independently of eliminating the circular dependency of rules_python on itself. This would allow us to eliminate subpar from our user-facing dependencies.

@alexeagle
Copy link
Collaborator

rules_python 0.1.0 has been released which upstreams the rules_python_external repo. Please switch from pip_import to pip_install which doesn't have this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants