Add hermetic_cc_toolchain for a hermetic cc toolchain #12135

prestonvanloon · 2023-03-15T21:07:23Z

What type of PR is this?

Other

What does this PR do? Why is it needed?

This PR introduces a replacement C/C++ toolchain for Prysm. Previously, we were using a LLVM toolchain from github.com/grailbio/bazel-toolchain and only when the user requested --config=llvm.

This PR aims to solve two problems:

Building C++ Depends on Host Tooling -- When building anything that depends on C or C++ code in prysm, bazel will use the host machine gcc. This can cause undesired behavior where identical builds on different linux machines result in different outputs. Namely, if your host is using a recent version of gcc then non-backwards compatible glibc libraries will be requested. See #7750 for an example. This behavior is non-hermetic and violates the desired principal of reproducible builds.

llvm Config Doesn't Work on Many OSes -- The way that the existing cc toolchain works is that it downloads an appropriate version of llvm based on the host's OS. Some users have complained that the LLVM toolchain fails when using an operating system like Pop!OS or even Arch Linux. Again, this may be non-hermetic behavior as the host's OS should not influence how the CC toolchain compiles Prysm.

The key features/changes are:

Default to using this cc_toolchain instead of the host / gcc toolchain
Pin glibc to 2.31 to support ubuntu 20.04 explicitly. (EOL is April, 2025)
Improve cross compilation by having a cc_toolchain that "just works". Enables support for multi-arch compilation for docker: Multiarch docker containers #12428

Which issues(s) does this PR fix?

Supercedes #12100

Fixes #7750
Fixes #9515

Other notes for review

Testing plan

To test this PR, we want to see that the release assets will build and everything else should build, but some manual targets are expected to fail on some configurations. (Some things are linux only)

# Build release assets
bazel build --config=release //cmd/beacon-chain //cmd/validator //cmd/client-stats //cmd/prysmctl

# Everything should work
bazel build //...

# Optionally, build everything including manual tests
bazel build //... --build_manual_tests

Builds on linux amd64
Builds on linux arm64 (untested)
Builds on darwin amd64 (Mac OS without M1/M2 chip) ❌ Currently failing
Builds on darwin arm64 (Mac OS with M1/M2 chip) ❌ Currently failing

❌ Builds on windows amd64 (Not supported)

⚠️ For darwin builds, I have added a hack that will allow a local C++ toolchain to be used until we can figure out how to hermetically provide a sysroot. We know this is possible with the docker cross compile, but we have to add that upstream to the toolchain provider. Additionally, there is a known issue where zig can't compile cgo on darwin. For the moment, we'll defer adding hermetic builds on darwin until those items can be resolved.

Additionally, we want to see cross compile continues to work on from linux amd64

bazel build --config=release --config=linux_arm64_docker  //cmd/beacon-chain //cmd/validator //cmd/client-stats //cmd/prysmctl
bazel build --config=release --config=osx_amd64_docker  //cmd/beacon-chain //cmd/validator //cmd/client-stats //cmd/prysmctl
bazel build --config=release --config=osx_arm64_docker  //cmd/beacon-chain //cmd/validator //cmd/client-stats //cmd/prysmctl
bazel build --config=release --config=windows_amd64_docker  //cmd/beacon-chain //cmd/validator //cmd/client-stats //cmd/prysmctl

Cross compile still works on linux amd64

…ig-cc

.bazelrc

james-prysm · 2023-05-23T14:29:19Z

.bazelrc

-build:llvm-asan --config=asan
-build:llvm-asan --linkopt -fuse-ld=ld.lld
-
-build:fuzz --@io_bazel_rules_go//go/config:tags=fuzz


just a note that we are removing fuzzing and the sanitize_address stuff is used for this.

james-prysm · 2023-05-23T14:29:49Z

build/bazelrc/zig-cc.bazelrc

+build --incompatible_enable_cc_toolchain_resolution
+
+# Add a no-op warning for users still using --config=llvm
+build:llvm --unconditional_warning="llvm config is no longer used as clang is now the default compiler"


we don't need to mention zig here at all right? even though it's using clang underneath?

kasey · 2023-05-23T14:30:02Z

WORKSPACE

+    "@zig_sdk//toolchain:linux_arm64_gnu.2.31",
+    "@zig_sdk//toolchain:darwin_amd64",
+    "@zig_sdk//toolchain:darwin_arm64",
+    "@zig_sdk//toolchain:windows_amd64",


+1 to your suggestion in huddle to rename this to match the upstream repo.

I can't rename this one, it's a static value here: https://github.com/uber/hermetic_cc_toolchain/blob/bbd965d3ab0803475bd3bb9b59fb50c3451b5c71/toolchain/defs.bzl#L79

kasey · 2023-05-23T14:30:36Z

build/bazelrc/zig-cc.bazelrc

@@ -0,0 +1,19 @@
+# From Bazel's perspective, this is almost equivalent to always specifying


another thing to rename zig-cc.bazelrc

kasey · 2023-05-23T14:31:38Z

build/bazelrc/zig-cc.bazelrc

+build --incompatible_enable_cc_toolchain_resolution
+
+# Add a no-op warning for users still using --config=llvm
+build:llvm --unconditional_warning="llvm config is no longer used as clang is now the default compiler"


will this force a build failure with llvm or just display a warning (that could get missed in the piles of compiler output)?

It is just a warning. Without some build:llvm, the build will fail entirely when using --config=llvm

prestonvanloon · 2023-05-23T15:58:37Z

darwin arm64 currently failing with

external/go_sdk/pkg/tool/darwin_arm64/link: running external/zig_sdk/tools/aarch64-macos-none/c++ failed: exit status 1
warning(link): framework not found for '-framework CoreFoundation'
warning(link): Framework search paths:
error: FrameworkNotFound

Looks like ziglang/zig#10513

prestonvanloon · 2023-05-23T16:24:50Z

Windows fails to build too. I think the bazel rules for protoc are not compatible... I'm not sure we ever supported building on windows.

…ig-cc

prestonvanloon · 2023-06-20T18:03:21Z

Another blocker for mac amd64 ziglang/zig#15438

…ig-cc

This reverts commit 7bb4b6b.

…ig-cc

james-prysm

was building fine locally so approving this

* Add bazel-zig-cc for a hermetic cc toolchain * gazelle * Remove llvm * remove wl * Add new URLs for renamed repo * gazelle * Update to v2.0.0-rc1 * bump to rc2 * Some PR feedback * use v2.0.0 from rc2 * Disable hermetic builds for mac and windows. * bump bazel version, add darwin hack * fix * Add the no-op emtpy cc toolchain code * typo and additional copy * update protobuf and fix vaticle warning * Revert "update protobuf and fix vaticle warning" This reverts commit 7bb4b6b. --------- Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

prestonvanloon added 8 commits March 15, 2023 15:58

Add bazel-zig-cc for a hermetic cc toolchain

9cf58cb

gazelle

845f57e

Remove llvm

08303b0

Merge branch 'develop' of github.com:prysmaticlabs/prysm into bazel-z…

2bb0073

…ig-cc

Merge branch 'develop' of github.com:prysmaticlabs/prysm into bazel-z…

01f62c0

…ig-cc

remove wl

421ba54

Add new URLs for renamed repo

a6766ea

Merge branch 'develop' of github.com:prysmaticlabs/prysm into bazel-z…

f55485e

…ig-cc

prestonvanloon marked this pull request as ready for review April 24, 2023 16:48

prestonvanloon requested a review from a team as a code owner April 24, 2023 16:49

prestonvanloon requested review from kasey, terencechain and rkapka April 24, 2023 16:49

prestonvanloon added 2 commits April 24, 2023 11:53

gazelle

4cbc1a4

Update to v2.0.0-rc1

a8606c2

prestonvanloon added the Ready For Review label Apr 24, 2023

prestonvanloon added 2 commits May 1, 2023 09:29

Merge branch 'develop' into bazel-zig-cc

f61fec3

Merge branch 'develop' into bazel-zig-cc

3b4705b

prestonvanloon changed the title ~~Add bazel-zig-cc for a hermetic cc toolchain~~ Add hermetic_cc_toolchain for a hermetic cc toolchain May 2, 2023

prestonvanloon added the OK to merge label May 2, 2023

prylabs-bulldozer bot added 10 commits May 3, 2023 04:34

Merge refs/heads/develop into bazel-zig-cc

3d23e8e

Merge refs/heads/develop into bazel-zig-cc

7fe5aa5

Merge refs/heads/develop into bazel-zig-cc

db9dd87

Merge refs/heads/develop into bazel-zig-cc

473b165

Merge refs/heads/develop into bazel-zig-cc

1780382

Merge refs/heads/develop into bazel-zig-cc

15468fd

Merge refs/heads/develop into bazel-zig-cc

62d8a44

Merge refs/heads/develop into bazel-zig-cc

9eda6b8

Merge refs/heads/develop into bazel-zig-cc

90d6792

Merge refs/heads/develop into bazel-zig-cc

0762e53

kasey reviewed May 23, 2023

View reviewed changes

.bazelrc Show resolved Hide resolved

james-prysm reviewed May 23, 2023

View reviewed changes

kasey reviewed May 23, 2023

View reviewed changes

Some PR feedback

fdfc683

kasey reviewed May 23, 2023

View reviewed changes

Merge branch 'develop' of github.com:prysmaticlabs/prysm into bazel-z…

b7e9c1a

…ig-cc

prestonvanloon added 10 commits June 20, 2023 13:05

use v2.0.0 from rc2

92375d0

Merge branch 'develop' of github.com:prysmaticlabs/prysm into bazel-z…

571a4a8

…ig-cc

Disable hermetic builds for mac and windows.

9ba3750

bump bazel version, add darwin hack

fa3ef31

fix

5bcb909

Add the no-op emtpy cc toolchain code

ec4c18c

typo and additional copy

aade909

update protobuf and fix vaticle warning

7bb4b6b

Revert "update protobuf and fix vaticle warning"

4bb6008

This reverts commit 7bb4b6b.

Merge branch 'develop' of github.com:prysmaticlabs/prysm into bazel-z…

6f39389

…ig-cc

james-prysm approved these changes Jun 21, 2023

View reviewed changes

kasey approved these changes Jun 22, 2023

View reviewed changes

prestonvanloon added the OK to merge label Jun 26, 2023

Merge branch 'develop' into bazel-zig-cc

b16b8de

prylabs-bulldozer bot merged commit a190440 into develop Jun 26, 2023

prylabs-bulldozer bot deleted the bazel-zig-cc branch June 26, 2023 14:31

prestonvanloon mentioned this pull request Jul 24, 2023

Bazel toolchain doesn't correctly figure out LLVM name for Suse distributions #9515

Closed

prestonvanloon mentioned this pull request Sep 26, 2023

Use bazel provided c++ toolchain #12100

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hermetic_cc_toolchain for a hermetic cc toolchain #12135

Add hermetic_cc_toolchain for a hermetic cc toolchain #12135

prestonvanloon commented Mar 15, 2023 •

edited

Loading

james-prysm May 23, 2023

james-prysm May 23, 2023

kasey May 23, 2023

prestonvanloon May 23, 2023

kasey May 23, 2023

kasey May 23, 2023

prestonvanloon May 23, 2023

prestonvanloon commented May 23, 2023

prestonvanloon commented May 23, 2023

prestonvanloon commented Jun 20, 2023

james-prysm left a comment

		@@ -0,0 +1,19 @@
		# From Bazel's perspective, this is almost equivalent to always specifying

Add hermetic_cc_toolchain for a hermetic cc toolchain #12135

Add hermetic_cc_toolchain for a hermetic cc toolchain #12135

Conversation

prestonvanloon commented Mar 15, 2023 • edited Loading

james-prysm May 23, 2023

Choose a reason for hiding this comment

james-prysm May 23, 2023

Choose a reason for hiding this comment

kasey May 23, 2023

Choose a reason for hiding this comment

prestonvanloon May 23, 2023

Choose a reason for hiding this comment

kasey May 23, 2023

Choose a reason for hiding this comment

kasey May 23, 2023

Choose a reason for hiding this comment

prestonvanloon May 23, 2023

Choose a reason for hiding this comment

prestonvanloon commented May 23, 2023

prestonvanloon commented May 23, 2023

prestonvanloon commented Jun 20, 2023

james-prysm left a comment

Choose a reason for hiding this comment

prestonvanloon commented Mar 15, 2023 •

edited

Loading