-
Notifications
You must be signed in to change notification settings - Fork 468
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Loading status checks…
bazel: Better remote caching (#29367)
~~This PR improves our remote caching by changing the way we use [`--workspace_status_command`](https://bazel.build/reference/command-line-reference#flag--workspace_status_command) which enables us to fetch 100% of actions from the remote cache when applicable.~~ ~~Previously I noticed CI jobs would always appear to rebuild the leafs of our repo (e.g. `environmentd` and `clusterd`), which unfortunately take the longest because of linking. This was because our workspace status command required generating some Rust code which was breaking our remote caching. Now with this change I've observed the remote caching working exactly as a local cache, e.g. a change in the `sql` crate only requires re-building `environmentd` whereas it used to also cause a recompile of `clusterd` and some other crates.~~ Edit: Unfortunately there is no free lunch and my testing was flawed. The original approach from this PR of tweaking the way we use `workspace_status_command` did not work, and we were actually just providing a constant value for the git hash instead of the updated value. After investigating further it turns out that Bazel treats local cache artifacts differently than remote cache artifacts when it comes to the workspace status and specificallly the `volatile-status.txt` file we rely on here, see bazelbuild/bazel#10075. The only way to prevent `build_info!()` from causing unnecessary rebuilds _and_ update the git hash for anything that does get rebuilt, is to side channel the necessary information like we're doing here. The approach is we write the current git hash to a known location, in this case `/tmp/mz_git_hash.txt` and then read it via Rust. `bin/bazel` has been updated to write this file, and I'll make the same change to `mzbuild`. When running with `--config=release-stamp` we fallback to the original behavior with `workspace_status_command` and formally "stamp" the builds which causes everything to get rebuilt. This is definitely a hack but from what I can tell there is no other way to get the same behavior. An alternative is we always just return a constant git hash when not running with `--config=release-stamped`, not sure how important stamping development builds is. ### Motivation Progress towards https://github.com/MaterializeInc/materialize/issues/26796 * Improve remote caching speed ### Checklist - [x] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [x] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [x] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [x] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [x] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.
- v0.127.1
- v0.127.0
- v0.126.5
- v0.126.4
- v0.126.3
- v0.126.2
- v0.126.1
- v0.126.0
- v0.125.3
- v0.125.2
- v0.125.1
- v0.125.0
- v0.124.2
- v0.124.1
- v0.124.0
- v0.123.1
- v0.123.0
- v0.122.2
- v0.122.1
- v0.122.0
- v0.121.2
- v0.121.1
- v0.121.0
- v0.120.3
- v0.120.2
- v0.120.1
- v0.120.0
- v0.119.3
- v0.119.2
- v0.119.1
- v0.119.0
- v0.118.3
- v0.118.2
- v0.118.1
- v0.118.0
- v0.117.3
- v0.117.2
- v0.117.1
- v0.117.0
- v0.116.0
Showing
9 changed files
with
86 additions
and
33 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters