Building bb_runner and bb_worker fails on Mac M1/arm64

[YngveNPettersen]

I have been trying to build bb_runner and bb_worker on an arm64 Mac M1 machine, but it fails due to not being able to resolve a dependency for nodejs_darwin_arm64 (I have previously unsuccessfully attempted to cross-compile for arm64, but only got x64 executables)

Bazel version is 4.2.1

Local installations of Nodejs tested have been both arm64 capable v12 and v16

AFAICT the problem was introduced by an npm dependency in bb_storage, although the root cause is probably in one of the nodejs/npm dependencies.

INFO: Repository npm instantiated at:
  <workdir>/WORKSPACE:175:12: in <toplevel>
  <storage>/external/build_bazel_rules_nodejs/index.bzl:78:17: in npm_install
Repository rule npm_install defined at:
  <storage>/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:654:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'npm':
   Traceback (most recent call last):
        File "<storage>//external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 556, column 31, in _npm_install_impl
                node = repository_ctx.path(get_node_label(repository_ctx))
Error in path: Unable to load package for @nodejs_darwin_arm64//:bin/node: The repository '@nodejs_darwin_arm64' could not be resolved
ERROR: Error fetching repository: Traceback (most recent call last):
        File "<storage>//external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 556, column 31, in _npm_install_impl
                node = repository_ctx.path(get_node_label(repository_ctx))
Error in path: Unable to load package for @nodejs_darwin_arm64//:bin/node: The repository '@nodejs_darwin_arm64' could not be resolved
ERROR: <storage>/external/com_github_buildbarn_bb_storage/pkg/otel/BUILD.bazel:53:8: @com_github_buildbarn_bb_storage//pkg/otel:stylesheet depends on @npm//purgecss/bin:purgecss in repository @npm which failed to fetch. no such package '@npm//purgecss/bin': Unable to load package for @nodejs_darwin_arm64//:bin/node: The repository '@nodejs_darwin_arm64' could not be resolved
ERROR: Analysis of target '//cmd/bb_runner:bb_runner' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.220s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 3 targets configured)

[EdSchouten]

Have you tried upgrading rules_nodejs? If that doesn’t work, please file an issue against rules _nodejs.

[YngveNPettersen]

This was a checkout of most recent master from the repo (49b2d95), no changes.

Also, the only real option seems to an update to 4.x of that rule (the current is 3.7, the newest is 4.4.1), and 4.0.0 mentions a number of breaking changes. I have absolutely no idea how relevant (or not) any of those are for my build targets.

<later ...> Conclusion: changing bb-remote-execution's WORKSPACE "build_bazel_rules_nodejs" to point at 4.4.1 works just as badly as 3.7.0; same errors

[EdSchouten]

Does a change along these lines work?

diff --git a/WORKSPACE b/WORKSPACE
index 69b3596..cfa7cb8 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -154,11 +154,15 @@ http_archive(
 
 http_archive(
     name = "build_bazel_rules_nodejs",
-    sha256 = "8f5f192ba02319254aaf2cdcca00ec12eaafeb979a80a1e946773c520ae0a2c9",
-    urls = ["https://github.com/bazelbuild/rules_nodejs/releases/download/3.7.0/rules_nodejs-3.7.0.tar.gz"],
+    sha256 = "4501158976b9da216295ac65d872b1be51e3eeb805273e68c516d2eb36ae1fbb",
+    urls = ["https://github.com/bazelbuild/rules_nodejs/releases/download/4.4.1/rules_nodejs-4.4.1.tar.gz"],
 )
 
-load("@build_bazel_rules_nodejs//:index.bzl", "npm_install")
+load("@build_bazel_rules_nodejs//:index.bzl", "node_repositories", "npm_install")
+
+node_repositories(
+    node_version = "16.6.1",
+)
 
 npm_install(
     name = "npm",

The default for rules_nodejs is to use NodeJS v12.13.0, which is too old to run on Mac M1. rules_nodejs does provide checksums for NodeJS for Mac M1 starting with v16, so we can explicitly request that for the time being.

[YngveNPettersen]

Thanks for that suggestion.

The build's dependency gathering did get further, but eventually failed with this message:

DEBUG: <storage>/external/bazel_gazelle/internal/go_repository.bzl:189:18: com_google_cloud_go: gazelle: finding module path for import google.golang.org/api/gensupport: go get: module google.golang.org/api@upgrade found (v0.59.0), but does not contain package google.golang.org/api/gensupport
gazelle: finding module path for import google.golang.org/api/gensupport: go get: module google.golang.org/api@upgrade found (v0.59.0), but does not contain package google.golang.org/api/gensupport
INFO: Analyzed 2 targets (417 packages loaded, 10823 targets configured).
INFO: Found 2 targets...
ERROR: <storage>/external/com_google_protobuf/BUILD:130:11: Compiling src/google/protobuf/extension_set.cc failed: I/O exception during sandboxed execution: com.google.devtools.build.lib.shell.ExecFailedException: java.io.IOException: Cannot run program "<tmp>/install/71ed47cad951a20fff87381f54639763/xcode-locator": error=86, Bad CPU type in executable
INFO: Elapsed time: 150.675s, Critical Path: 0.31s
INFO: 21 processes: 20 internal, 1 darwin-sandbox.
FAILED: Build did NOT complete successfully

[EdSchouten]

Do you have Rosetta installed?

[YngveNPettersen]

No, I have for the time being decided to avoid that, to make sure I am actually running arm64 executables (as mentioned above, an attempt to cross-compile only produced x64 executables, not arm64 ones)

[EdSchouten]

Then there’s little that I can do here, as the xcode-locator tool that causes this error is part of Bazel itself. Looks like Bazel for M1 depends on Rosetta.

[YngveNPettersen]

When building a native version, Rosetta should not be necessary.

I don't know if the issue is caused by Bazel, or not; but the installed version on the system is the M1 Arm64 version.

That the failing file is in a temp dir indicates to me that it is part of one of the dependencies, not Bazel.

Additionally, there seems to be at least one report about not being able to build protocol for M1, protocolbuffers/protobuf#8428 , although v3.17.3 is supposed to support M1 according to the changelog (3.19.0 is the newest version).

[EdSchouten]

You could try to build a copy of Bazel that has this change applied:

diff --git a/tools/osx/BUILD b/tools/osx/BUILD
index d2c0fe618d..d7cd27a6b1 100644
--- a/tools/osx/BUILD
+++ b/tools/osx/BUILD
@@ -29,7 +29,7 @@ exports_files([
 
 DARWIN_XCODE_LOCATOR_COMPILE_COMMAND = """
   /usr/bin/xcrun --sdk macosx clang -mmacosx-version-min=10.9 -fobjc-arc -framework CoreServices \
-      -framework Foundation -o $@ $<
+      -framework Foundation -arch arm64 -arch x86_64 -o $@ $<
 """
 
 genrule(

That should cause it to include a copy of xcode-locator that is multi-arch.

[YngveNPettersen]

I didn't build the install from source, so that will require some work.

I did find this bazel bug that seems to be related: bazelbuild/bazel#13944

[YngveNPettersen]

Just tried again to build the runner and worker on M1, it now fails in a different dependency:

ERROR: <stagedir>/external/com_google_protobuf/BUILD:357:11: Compiling src/google/protobuf/compiler/command_line_interface.cc failed: I/O exception during sandboxed execution: com.google.devtools.build.lib.shell.ExecFailedException: java.io.IOException: Cannot run program "<tmpdir>/xcode-locator": error=86, Bad CPU type in executable

[EdSchouten]

Sure. But there's nothing I can do about that issue from our end. I've filed a PR against Bazel to get xcode-locator built as a universal binary, but that's all we can do from Buildbarn's perspective.

[YngveNPettersen]

I followed that, and as that was resolved (although there is no new bazel version release) I decided to retry; and since it is no longer failing in NPM, but protobuf, I decided to update the status

[Gormo]

@YngveNPettersen odd that you're getting issues with that. I built yesterday on my MacBook Air M1 without issues using the bazel-version provided by bazelisk.

Maybe you need to update your Xcode tools? I'm using the latest version and OS X Monterey

[EdSchouten]

@Gormo No, @YngveNPettersen doesn't have Rosetta installed. If you have Rosetta, then the existing xcode-locator binary should work all right.

[Gormo]

@EdSchouten ok, that could be it. I didn't think I had Rosetta installed but maybe I have. Will need to check later tonight.