Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support reproducible builds #176

Closed
Monnoroch opened this issue Apr 15, 2019 · 4 comments · Fixed by #373
Closed

Support reproducible builds #176

Monnoroch opened this issue Apr 15, 2019 · 4 comments · Fixed by #373
Assignees

Comments

@Monnoroch
Copy link

Monnoroch commented Apr 15, 2019

rules_python allows using a requirements.txt file with pinned down versions of dependencies. However, this is not good enough, since the actual archive downloaded can be different each time, which is really bad for Bazel's aggressive caching strategy and for cross-machine reproducibility. Plus, this it totally in spirit of Bazel. I fully realize that this feature will require explicitly listing transitive dependencies but it is totally acceptable when maximum reproducibility is required. The way I would propose to implement it would be a generator that given an input requirements.txt file outputs a list of targets to be added to the WORKSPACE file. These rules should also include sha 256 hashes of compiled wheels or other artifacts. Another important thing to remember is cross-compilation: some Python libraries (such as numpy) include native code and require to be built for a target system. The above generator should support common patterns and be extensible enough for developers to handle tricky cases. Right now, these rules download different files on Windows, so with a linux-only CI environment I experience issues when people that use Windows try to build code.

Related: #140.

Somewhat related: #154.

@alexeagle
Copy link
Collaborator

rules_python 0.1.0 has been released which upstreams the rules_python_external repo. Please switch from pip_import to pip_install which doesn't have this issue.

@Monnoroch
Copy link
Author

Monnoroch commented Oct 16, 2020

Could you please clarify how exactly is pip_install reproducible? I see that it's still using requirements.txt which means neither we specify sha256 hash for the downloaded archives nor we pin transitive dependencies. From the looks of it, nothing has really changed as far as hermeticity goes, I will still occasionally get different archives downloaded after bazel clean --expunge.

@alexeagle alexeagle reopened this Oct 16, 2020
@alexeagle
Copy link
Collaborator

I closed this based on the README from rules_python_external.

I agree though, you need an extra bit of pip-compile or equivalent to pin and lock dependencies. I'll send a PR to add that rule.

alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Oct 19, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Oct 19, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Oct 26, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Oct 26, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Oct 26, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Oct 26, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Oct 26, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Oct 26, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Nov 2, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Nov 2, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Nov 2, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
cryptotraderdeku pushed a commit to cryptotraderdeku/rules_python that referenced this issue Jan 17, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
@github-actions
Copy link

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

@github-actions github-actions bot added the Can Close? Will close in 30 days if there is no new activity label Apr 14, 2021
@thundergolfer thundergolfer removed the Can Close? Will close in 30 days if there is no new activity label Apr 19, 2021
@thundergolfer thundergolfer self-assigned this Apr 19, 2021
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Apr 20, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Apr 20, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Apr 20, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Apr 20, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Apr 20, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Apr 20, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Apr 20, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Apr 20, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue Apr 21, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue May 7, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle pushed a commit to alexeagle/rules_python that referenced this issue May 7, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes bazelbuild#176
alexeagle added a commit that referenced this issue May 7, 2021
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes #176
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants