This repository has been archived by the owner on Oct 2, 2023. It is now read-only.
Don't upload large files to remote-cache #1856
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gravypod
approved these changes
May 21, 2021
alexeagle
added a commit
to alexeagle/rules_docker
that referenced
this pull request
May 24, 2021
Similar to bazelbuild#1856. I found these by patching that PR and running through a big monorepo and searching execution log for more remote cache uploads over 10MB
alexeagle
added a commit
to alexeagle/rules_docker
that referenced
this pull request
May 24, 2021
Similar to bazelbuild#1856. I found these by patching that PR and running through a big monorepo and searching execution log for more remote cache uploads over 10MB
alexeagle
added a commit
that referenced
this pull request
May 24, 2021
Similar to #1856. I found these by patching that PR and running through a big monorepo and searching execution log for more remote cache uploads over 10MB
|
Is there a way to disable this? We really do want to use remote-cache for our bazel Docker builds, our cache is fast and building locally each time takes way too long. |
|
Which action is slow? Join_layers? Maybe we should just revert that one. |
|
Yes. I think reverting that one is a good idea. Our bazel cache is a giant bucket and we'd prefer not to rebuild anything if possible; can't see a way to force the join_layers to cache if this flag is set. At the very least, you could always add the tag I also noticed that the apt rules also have remote-cache disabled, these are also slow and should also be cached IMO. Either that, or some way to prevent the rebuild of Thanks for the prompt response. |
alexeagle
added a commit
that referenced
this pull request
Mar 16, 2022
These were introduced to reduce load on a remote-cache instance to avoid network saturation. A month later, a feature was added in one remote-cache implementatation which provides a different fix: buchgr/bazel-remote#440 rejects large input files on upload. In practice, while these action do often produce huge outputs, they are also slow to re-execute. In many cases it's worth it to use a remote-cache for RunAndCommitLayer in particular to avoid a local rebuild even though it's a large network fetch. Currently users can't configure this because we've hardcoded the values. If they do want to keep the no-remote-cache execution requirement, they can do this via a tag (provided they opt-in to experimental_allow_tags_propagation, see bazelbuild/bazel#8830) #1856 (comment) is an example of a user asking for these to be removed.
alexeagle
added a commit
that referenced
this pull request
Mar 16, 2022
These were introduced to reduce load on a remote-cache instance to avoid network saturation. A month later, a feature was added in one remote-cache implementatation which provides a different fix: buchgr/bazel-remote#440 rejects large input files on upload. In practice, while these action do often produce huge outputs, they are also slow to re-execute. In many cases it's worth it to use a remote-cache for RunAndCommitLayer in particular to avoid a local rebuild even though it's a large network fetch. Currently users can't configure this because we've hardcoded the values. If they do want to keep the no-remote-cache execution requirement, they can do this via a tag (provided they opt-in to experimental_allow_tags_propagation, see bazelbuild/bazel#8830) #1856 (comment) is an example of a user asking for these to be removed.
gravypod
pushed a commit
that referenced
this pull request
Mar 16, 2022
These were introduced to reduce load on a remote-cache instance to avoid network saturation. A month later, a feature was added in one remote-cache implementatation which provides a different fix: buchgr/bazel-remote#440 rejects large input files on upload. In practice, while these action do often produce huge outputs, they are also slow to re-execute. In many cases it's worth it to use a remote-cache for RunAndCommitLayer in particular to avoid a local rebuild even though it's a large network fetch. Currently users can't configure this because we've hardcoded the values. If they do want to keep the no-remote-cache execution requirement, they can do this via a tag (provided they opt-in to experimental_allow_tags_propagation, see bazelbuild/bazel#8830) #1856 (comment) is an example of a user asking for these to be removed.
|
Thank you! 😊 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1855