Skip to content

Commit

Permalink
bazel: Add support for darwin arm64 (M1 builds)
Browse files Browse the repository at this point in the history 
We've been using the bazel-toolchain library so we can use our custom LLVM
toolchain. This was working perfectly on CI because we compile on Linux there.

On our local M1 machines this was falling back on our xcode compiler instead of
the custom toolchain because bazel-toolchain doesn't have support for the
darwin-arm64 configuration. There is an existing GitHub issue about this
(bazel-contrib#88) with a lot of activity
but nothing landed. There also seemed to be some outstanding PRs, but they are
complete rewrites; and it's not clear when they would land. My guess is that we
may be able to drop our patches at some point, but we shouldn't wait for it.

The main changes here are as follows:

* We add support for a darwin-arm64 toolchain to bazel-toolchain.

* This requires some additional configuration to handle the various ridiculous
goop that macOS builds need; bazel already has toolchain support for this, so
most of it is copied from the bazelbuild repo. It would be great to upstream
this, but that might be a significant effort.

* It appears that bazel-toolchain handled everything using the
`unix_cc_toolchain_config`, which only had limited supported for Objective C.
Since we need to fully be able to compile objective C applications, we need to
include `osx_cc_toolchain_config` as well.

* The `osx_cc_toolchain_config` uses a clang wrapper called `wrapped_clang` and
`wrapped_clang_pp`. This is written as a C++ source file for some reason, so it
needs to be compiled as part of the toolchain build. This wrapper forwards to
xcode clang/clang++ by default, but we want to forward to our own toolchain
instead. So we have to modify the wrapper slightly to look at some environment
variables to determine where the real toolchain is.

* The clang compiler includes some directories by default, specifically ones
that are included with the compiler. This works if you run the toolchain
manually, but bazel doesn't like this because it wants to make sure that it
knows about all included headers. Normally you should be able to use the
`cxx_builtin_include_directories` attribute to notify bazel about these
directories, but it doesn't seem to work. It seems that the problem may be that
the included paths are absolute instead of relative, so they don't appear to be
the same path to bazel. Using clang's `-no-canonical-prefixes` flag is meant to
fix this, but it didn't seem to work for me. As an alternate workaround, I made
sure to include all the built-in directories again via `-isystem` so that the
paths match exactly.
  • Loading branch information
@binji
binji committed Sep 28, 2022
1 parent cf65d07 commit 9e8842b
Show file tree
Hide file tree
Showing 8 changed files with 3,624 additions and 38 deletions.
8 changes: 8 additions & 0 deletions platforms/BUILD.bazel
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,11 @@ platform(
"@platforms//cpu:x86_64",
],
)

platform(
name = "darwin-arm64",
constraint_values = [
"@platforms//os:osx",
"@platforms//cpu:arm64",
],
)
1 change: 1 addition & 0 deletions toolchain/BUILD.toolchain.tpl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ filegroup(
filegroup(
name = "internal-use-wrapped-tools",
srcs = [
%{wrapped_clang_tools}
"bin/cc_wrapper.sh",
"bin/host_libtool_wrapper.sh",
],
Expand Down
97 changes: 64 additions & 33 deletions toolchain/cc_toolchain_config.bzl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.

load(
"//toolchain/internal:osx_cc_toolchain_config.bzl",
osx_cc_toolchain_config = "cc_toolchain_config",
)
load(
"//toolchain/internal:unix_cc_toolchain_config.bzl",
unix_cc_toolchain_config = "cc_toolchain_config",
Expand Down Expand Up @@ -60,10 +64,19 @@ def cc_toolchain_config(
abi_version,
abi_libc_version,
) = {
"darwin-arm64": (
"clang-arm64-darwin",
"arm64-apple-macosx",
"darwin_arm64",
"macosx",
"clang",
"darwin_arm64",
"darwin_arm64",
),
"darwin-x86_64": (
"clang-x86_64-darwin",
"x86_64-apple-macosx",
"darwin",
"darwin_x86_64",
"macosx",
"clang",
"darwin_x86_64",
Expand Down Expand Up @@ -235,13 +248,13 @@ def cc_toolchain_config(

# C++ built-in include directories:
cxx_builtin_include_directories = []
if toolchain_path_prefix.startswith("/"):
cxx_builtin_include_directories.extend([
toolchain_path_prefix + "include/c++/v1",
toolchain_path_prefix + "include/{}/c++/v1".format(target_system_name),
toolchain_path_prefix + "lib/clang/{}/include".format(llvm_version),
toolchain_path_prefix + "lib64/clang/{}/include".format(llvm_version),
])
compiler_include_directories = [
toolchain_path_prefix + "include/c++/v1",
toolchain_path_prefix + "include/{}/c++/v1".format(target_system_name),
toolchain_path_prefix + "lib/clang/{}/include".format(llvm_version),
toolchain_path_prefix + "lib64/clang/{}/include".format(llvm_version),
]
cxx_builtin_include_directories.extend(compiler_include_directories)

sysroot_path = compiler_configuration["sysroot_path"]
sysroot_prefix = ""
Expand Down Expand Up @@ -333,28 +346,46 @@ def cc_toolchain_config(
unfiltered_compile_flags = _fmt_flags(compiler_configuration["unfiltered_compile_flags"], toolchain_path_prefix)

# Source: https://cs.opensource.google/bazel/bazel/+/master:tools/cpp/unix_cc_toolchain_config.bzl
unix_cc_toolchain_config(
name = name,
cpu = target_cpu,
compiler = compiler,
toolchain_identifier = toolchain_identifier,
host_system_name = host_system_name,
target_system_name = target_system_name,
target_libc = target_libc,
abi_version = abi_version,
abi_libc_version = abi_libc_version,
cxx_builtin_include_directories = cxx_builtin_include_directories,
tool_paths = tool_paths,
compile_flags = compile_flags,
dbg_compile_flags = dbg_compile_flags,
opt_compile_flags = opt_compile_flags,
cxx_flags = cxx_flags,
link_flags = link_flags,
link_libs = link_libs,
opt_link_flags = opt_link_flags,
unfiltered_compile_flags = unfiltered_compile_flags,
coverage_compile_flags = coverage_compile_flags,
coverage_link_flags = coverage_link_flags,
supports_start_end_lib = supports_start_end_lib,
builtin_sysroot = sysroot_path,
)
if host_os == "darwin":
osx_cc_toolchain_config(
name = name,
cpu = target_cpu,
compiler = compiler,
cxx_builtin_include_directories = cxx_builtin_include_directories,
tool_paths_overrides = tool_paths,
builtin_sysroot = sysroot_path,

# Added so wrapped_clang and wrapped_clang_pp can forward to our
# clang instead of the xcode clang.
clang_path = toolchain_path_prefix + "bin/clang",
clang_pp_path = toolchain_path_prefix + "bin/clang++",
# The builtin include directories also need to be included with
# -isystem to work properly with bazel.
compiler_include_directories = compiler_include_directories,
)
else:
unix_cc_toolchain_config(
name = name,
cpu = target_cpu,
compiler = compiler,
toolchain_identifier = toolchain_identifier,
host_system_name = host_system_name,
target_system_name = target_system_name,
target_libc = target_libc,
abi_version = abi_version,
abi_libc_version = abi_libc_version,
cxx_builtin_include_directories = cxx_builtin_include_directories,
tool_paths = tool_paths,
compile_flags = compile_flags,
dbg_compile_flags = dbg_compile_flags,
opt_compile_flags = opt_compile_flags,
cxx_flags = cxx_flags,
link_flags = link_flags,
link_libs = link_libs,
opt_link_flags = opt_link_flags,
unfiltered_compile_flags = unfiltered_compile_flags,
coverage_compile_flags = coverage_compile_flags,
coverage_link_flags = coverage_link_flags,
supports_start_end_lib = supports_start_end_lib,
builtin_sysroot = sysroot_path,
)
2 changes: 1 addition & 1 deletion toolchain/internal/common.bzl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

SUPPORTED_TARGETS = [("linux", "x86_64"), ("linux", "aarch64"), ("darwin", "x86_64")]
SUPPORTED_TARGETS = [("linux", "x86_64"), ("linux", "aarch64"), ("darwin", "x86_64"), ("darwin", "arm64")]

host_tool_features = struct(
SUPPORTS_ARG_FILE = "supports_arg_file",
Expand Down
117 changes: 115 additions & 2 deletions toolchain/internal/configure.bzl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,96 @@ load(
_sysroot_path = "sysroot_path",
)

# FIGMA: Copied `_compile_cc_file_single_arch` and `_compile_cc_file` from
# https://github.com/bazelbuild/bazel/blob/5.2.0/tools/cpp/osx_cc_configure.bzl
#
# This is used to compile `wrapped_clang` and `wrapped_clang_pp`. These are
# tools that the regular bazel xcode build uses. See `wrapped_clang.cc` for
# more info. We have to make some small changes so that we can redirect to our
# clang instead of the one in the xcode toolchain.

# TODO: Remove once Xcode 12 is the minimum supported version
def _compile_cc_file_single_arch(repository_ctx, src_name, out_name):
env = repository_ctx.os.environ
xcrun_result = repository_ctx.execute([
"env",
"-i",
"DEVELOPER_DIR={}".format(env.get("DEVELOPER_DIR", default = "")),
"xcrun",
"--sdk",
"macosx",
"clang",
"-mmacosx-version-min=10.9",
"-std=c++11",
"-lc++",
"-O3",
"-o",
out_name,
src_name,
], 30)
if (xcrun_result.return_code != 0):
error_msg = (
"return code {code}, stderr: {err}, stdout: {out}"
).format(
code = xcrun_result.return_code,
err = xcrun_result.stderr,
out = xcrun_result.stdout,
)
fail(out_name + " failed to generate. Please file an issue at " +
"https://github.com/bazelbuild/bazel/issues with the following:\n" +
error_msg)

def _compile_cc_file(repository_ctx, src_name, out_name):
env = repository_ctx.os.environ
xcrun_result = repository_ctx.execute([
"env",
"-i",
"DEVELOPER_DIR={}".format(env.get("DEVELOPER_DIR", default = "")),
"xcrun",
"--sdk",
"macosx",
"clang",
"-mmacosx-version-min=10.9",
"-std=c++11",
"-lc++",
"-arch",
"arm64",
"-arch",
"x86_64",
"-Wl,-no_adhoc_codesign",
"-Wl,-no_uuid",
"-O3",
"-o",
out_name,
src_name,
], 30)

if xcrun_result.return_code == 0:
xcrun_result = repository_ctx.execute([
"env",
"-i",
"codesign",
"--identifier", # Required to be reproducible across archs
out_name,
"--force",
"--sign",
"-",
out_name,
], 30)
if xcrun_result.return_code != 0:
error_msg = (
"codesign return code {code}, stderr: {err}, stdout: {out}"
).format(
code = xcrun_result.return_code,
err = xcrun_result.stderr,
out = xcrun_result.stdout,
)
fail(out_name + " failed to generate. Please file an issue at " +
"https://github.com/bazelbuild/bazel/issues with the following:\n" +
error_msg)
else:
_compile_cc_file_single_arch(repository_ctx, src_name, out_name)

def _include_dirs_str(rctx, key):
dirs = rctx.attr.cxx_builtin_include_directories.get(key)
if not dirs:
Expand Down Expand Up @@ -159,6 +249,11 @@ def llvm_register_toolchains():
},
)

# FIGMA: Only include `wrapped_clang` for darwin builds.
wrapped_clang_tools_str = ''
if os == 'darwin':
wrapped_clang_tools_str = '''"wrapped_clang",\n"wrapped_clang_pp",\n'''

# BUILD file with all the generated toolchain definitions.
rctx.template(
"BUILD.bazel",
Expand All @@ -169,6 +264,7 @@ def llvm_register_toolchains():
"%{symlinked_tools}": symlinked_tools_str,
"%{llvm_repo_package}": _pkg_name_from_label(llvm_repo_label),
"%{host_dl_ext}": host_dl_ext,
"%{wrapped_clang_tools}": wrapped_clang_tools_str,
},
)

Expand All @@ -194,6 +290,14 @@ def llvm_register_toolchains():
},
)

# FIGMA: Copied from https://github.com/bazelbuild/bazel/blob/5.2.0/tools/cpp/osx_cc_configure.bzl
# See comment at `_compile_cc_file` above.
if os == "darwin":
wrapped_clang_src_path = rctx.path(Label("//toolchain:wrapped_clang.cc"))
_compile_cc_file(rctx, wrapped_clang_src_path, "wrapped_clang")
rctx.symlink("wrapped_clang", "wrapped_clang_pp")


def _cc_toolchains_str(
workspace_name,
toolchain_info,
Expand Down Expand Up @@ -275,6 +379,13 @@ def _cc_toolchain_str(
# them into `dict`s.
host_tools_info = json.decode(json.encode(host_tools_info))

# FIGMA: choose between `apple_cc_toolchain` and `cc_toolchain` based on
# the host os.
if host_os == 'darwin':
cc_toolchain = 'apple_cc_toolchain'
else:
cc_toolchain = 'cc_toolchain'

template = """
# CC toolchain for cc-clang-{suffix}.
Expand Down Expand Up @@ -324,7 +435,7 @@ toolchain(

if use_absolute_paths:
template = template + """
cc_toolchain(
{cc_toolchain}(
name = "cc-clang-{suffix}",
all_files = ":empty",
compiler_files = ":empty",
Expand Down Expand Up @@ -380,7 +491,7 @@ filegroup(name = "linker-files-{suffix}", srcs = [":linker-components-{suffix}"{
filegroup(name = "objcopy-files-{suffix}", srcs = ["{llvm_repo_label_prefix}objcopy"{extra_files_str}])
filegroup(name = "strip-files-{suffix}", srcs = ["{llvm_repo_label_prefix}strip"{extra_files_str}])
cc_toolchain(
{cc_toolchain}(
name = "cc-clang-{suffix}",
all_files = "all-files-{suffix}",
ar_files = "archiver-files-{suffix}",
Expand Down Expand Up @@ -424,4 +535,6 @@ cc_toolchain(
llvm_version = toolchain_info.llvm_version,
extra_files_str = extra_files_str,
host_tools_info = host_tools_info,

cc_toolchain = cc_toolchain, # FIGMA
)
Loading

0 comments on commit 9e8842b

Please sign in to comment.