High rate of spurious CI failures on macOS machines #14459

fmeum · 2021-12-21T08:10:31Z

Description of the problem / feature request:

I run daily CI checks in my rulesets' GitHub Actions pipeline. The macOS pipelines, running on macos-latest, fail every few days with two kinds of spurious failures that I have never been able to reproduce locally:

Issue 1:

Starting local Bazel server and connecting to it...
... still trying to connect to local Bazel server after 10 seconds ...
... still trying to connect to local Bazel server after 20 seconds ...
... still trying to connect to local Bazel server after 30 seconds ...
... still trying to connect to local Bazel server after 40 seconds ...
... still trying to connect to local Bazel server after 50 seconds ...
... still trying to connect to local Bazel server after 60 seconds ...
... still trying to connect to local Bazel server after 70 seconds ...
... still trying to connect to local Bazel server after 80 seconds ...
... still trying to connect to local Bazel server after 90 seconds ...
... still trying to connect to local Bazel server after 100 seconds ...
... still trying to connect to local Bazel server after 110 seconds ...
FATAL: couldn't connect to server (1753) after 120 seconds.
Error: Process completed with exit code 37.

Issue 2:

ERROR: /Users/runner/work/rules_jni/rules_jni/tests/libjvm_stub/BUILD.bazel:116:12: Target '//libjvm_stub:HelloFromJava' depends on toolchain '@local_config_cc//:cc-compiler-darwin', which cannot be found: error loading package '@local_config_cc//': cannot load '@local_config_cc_toolchains//:osx_archs.bzl': no such file'
ERROR: Analysis of target '//libjvm_stub:HelloFromJava' failed; build aborted: Analysis failed

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have no way to consistently reproduce this issue, but it happens every few days on rules_jni's CI schedule.

What operating system are you running Bazel on?

macOS 10.15

What's the output of `bazel info release`?

Over time, I have hit the issues on 4.2.2, 5.0.0rc3 and various last_green builds.

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

I can make arbitrary changes to the CI config if that helps to gather more information on the cause of these issues.

The text was updated successfully, but these errors were encountered:

thii · 2021-12-21T08:34:26Z

I believe it was waiting for xcode-locator, that can be super long if the machine has many Xcodes.

Try this hack: https://www.smileykeith.com/2021/03/08/locking-xcode-in-bazel/

fmeum · 2021-12-21T09:05:14Z

Thanks, I will try that. Do you think that this could also be related to Issue 2?

thii · 2021-12-21T11:02:12Z

It could be, since that file is only created if Xcode is successfully detected.

bazel/tools/cpp/cc_configure.bzl

Line 73 in b4b0c32

    
           repository_ctx.symlink(paths["@bazel_tools//tools/osx/crosstool:osx_archs.bzl"], "osx_archs.bzl")

sventiffe · 2022-01-14T09:19:13Z

@fmeum did @thii's suggestion fix the issue?

fmeum · 2022-01-14T09:29:03Z

I just checked again and found no new failures in the two weeks since I started using a checked-in xcode_version_config. I will close this now as the workaround seems to have worked and open a new issue if it should show up again.

fmeum · 2022-01-15T10:06:36Z

@sventiffe Reopening since the exact same issue appeared again today over at rules_jni.

I started pinning with --xcode_version_config in fmeum/rules_jni@b502180. Today, the runs on macos-latest with Bazel 4.0.0 and Bazel 5.0.0rc3 failed. I attached the CI logs, the failing step is here with the message (in the case of Bazel 5.0.0rc3 with bzlmod):

2022/01/15 00:35:14 Downloading https://releases.bazel.build/4.0.0/release/bazel-4.0.0-darwin-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 0e7ba86d-1a19-4b3a-a770-f0c3d58d166f
Loading: 
Loading: 0 packages loaded
Loading: 11 packages loaded
    currently loading: analysis/cc_jni_library ... (2 packages)
Loading: 11 packages loaded
    currently loading: analysis/cc_jni_library ... (2 packages)
Analyzing: 21 targets (13 packages loaded, 0 targets configured)
Analyzing: 21 targets (15 packages loaded, 1 target configured)
Analyzing: 21 targets (18 packages loaded, 8 targets configured)
Analyzing: 21 targets (18 packages loaded, 8 targets configured)
Analyzing: 21 targets (19 packages loaded, 8 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (38 packages loaded, 206 targets configured)
Analyzing: 21 targets (38 packages loaded, 206 targets configured)
Analyzing: 21 targets (38 packages loaded, 206 targets configured)
ERROR: /Users/runner/work/rules_jni/rules_jni/tests/native_loader/src/test/java/com/example/math/BUILD.bazel:3:17: Target '//native_loader/src/test/java/com/example/math:math_remove_this_part_' depends on toolchain '@local_config_cc//:cc-compiler-darwin', which cannot be found: error loading package '@local_config_cc//': cannot load '@local_config_cc_toolchains//:osx_archs.bzl': no such file'
ERROR: Analysis of target '//native_loader/src/test/java/com/example/math:math' failed; build aborted: Analysis failed
INFO: Elapsed time: 216.940s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (39 packages loaded, 206 targets configured)
ERROR: Couldn't start the build. Unable to run tests
FAILED: Build did NOT complete successfully (39 packages loaded, 206 targets configured)
Error: Process completed with exit code 1.

@thii Are there any additional diagnostics that I could enable that would help diagnose the underlying issue?

fmeum · 2022-02-10T22:15:41Z

@keith I just encountered another failure and attached the the logs. They include the content of the BUILD file, which turns out not to be empty.

keith · 2022-02-10T22:24:15Z

That file actually looks pretty good to me, since yours didn't fail with osx_archs import, it's because that file doesn't contain darwin https://github.com/bazelbuild/bazel/blob/master/tools/osx/crosstool/osx_archs.bzl

I'm not sure where that's coming from but it should be darwin_x86_64 probably instead

fmeum · 2022-02-11T15:37:49Z

That file actually looks pretty good to me, since yours didn't fail with osx_archs import, it's because that file doesn't contain darwin https://github.com/bazelbuild/bazel/blob/master/tools/osx/crosstool/osx_archs.bzl

I'm not sure where that's coming from but it should be darwin_x86_64 probably instead

Nice catch. I think I found the root cause and submitted a fix as #14796.

keith · 2022-02-11T17:34:34Z

Nice, I'll keep an eye on that. My assumption for why this is similar to the flaky CI case is because the toolchain only falls back to this codepath on macOS when Xcode cannot be found, which could happen in the case of a timeout running the discovery logic for that. This would likely workaround that issue if folks needed full Xcode in their cases https://www.smileykeith.com/2021/03/08/locking-xcode-in-bazel/

Previously, if the xcode_locator failed and cc_autoconf_toolchain used the non-Xcode C++ toolchain as a fallback, its reference to `@local_config_cc//:cc-compiler-darwin`, where darwin is the legacy cpu value for x86_64 macOS, would be invalid. Fixes #14459 Closes #14796. PiperOrigin-RevId: 451860477 Change-Id: Iec115f600ebb7ac0786b2169276d25e3ff5d54bf Co-authored-by: Fabian Meumertzheim <[email protected]>

sventiffe added platform: apple more data needed labels Jan 14, 2022

fmeum closed this as completed Jan 14, 2022

fmeum reopened this Jan 15, 2022

fmeum mentioned this issue Jan 20, 2022

bazel build @local_config_cc//... failed on macOS Big Sur #14603

Closed

keith mentioned this issue Jan 20, 2022

Reference osx_archs.bzl from bazel_tools #13529

Closed

cgrindel mentioned this issue Jan 20, 2022

Revert use of custom Bazel for macos_build in CI workflow cgrindel/rules_spm#112

Open

gregestren added the type: bug label Jan 31, 2022

fmeum mentioned this issue Feb 11, 2022

Define cc-compiler-darwin in Xcode toolchain #14796

Closed

copybara-service bot closed this as completed in 2d42925 May 30, 2022

ckolli5 mentioned this issue Jun 20, 2022

Define cc-compiler-darwin in Xcode toolchain #15709

Merged

dtolnay mentioned this issue Jun 19, 2023

Bazel CI on macOS is flaky dtolnay/cxx#1231

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

High rate of spurious CI failures on macOS machines #14459

High rate of spurious CI failures on macOS machines #14459

fmeum commented Dec 21, 2021 •

edited

Loading

thii commented Dec 21, 2021

fmeum commented Dec 21, 2021

thii commented Dec 21, 2021

sventiffe commented Jan 14, 2022

fmeum commented Jan 14, 2022

fmeum commented Jan 15, 2022 •

edited

Loading

fmeum commented Feb 10, 2022

keith commented Feb 10, 2022

fmeum commented Feb 11, 2022

keith commented Feb 11, 2022

High rate of spurious CI failures on macOS machines #14459

High rate of spurious CI failures on macOS machines #14459

Comments

fmeum commented Dec 21, 2021 • edited Loading

Description of the problem / feature request:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What operating system are you running Bazel on?

What's the output of bazel info release?

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

thii commented Dec 21, 2021

fmeum commented Dec 21, 2021

thii commented Dec 21, 2021

sventiffe commented Jan 14, 2022

fmeum commented Jan 14, 2022

fmeum commented Jan 15, 2022 • edited Loading

fmeum commented Feb 10, 2022

keith commented Feb 10, 2022

fmeum commented Feb 11, 2022

keith commented Feb 11, 2022

fmeum commented Dec 21, 2021 •

edited

Loading

What's the output of `bazel info release`?

fmeum commented Jan 15, 2022 •

edited

Loading