Bazel 4.0.0 breaks macOS Mojave support (for some XCode CLT versions)

[dhalperi]

Description of the problem / feature request:

Several folks on my team are reporting issues after upgrading to 4.0.0. One user is on 10.13.6 (High Sierra) and another is on 10.14.2 (Mojave). Both are able to build fine with 3.7.2, and both have build failures with libtool on 4.0.0:

ERROR: /private/var/tmp/_bazel_matt/c2cb640b277cd35f816879cfba9892ed/external/com_google_protobuf/BUILD:334:11: Linking external/com_google_protobuf/libprotoc_lib.a failed: (Exit 1): libtool failed: error executing command /usr/bin/libtool @bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/libprotoc_lib.a-2.params
Use --sandbox_debug to see verbose messages from the sandbox libtool failed: error executing command /usr/bin/libtool @bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/libprotoc_lib.a-2.params
Use --sandbox_debug to see verbose messages from the sandbox
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: no output file specified (specify with -o output)
Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -static [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-sacLT] [-no_warning_for_no_symbols]
Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-o output] [-install_name name] [-compatibility_version #] [-current_version #] [-seg1addr 0x#] [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table <filename>] [-seg_addr_table_filename <file_system_path>] [-all_load] [-noall_load]

From what I can tell, this is extremely reminiscent of #7397 -- seems related to AR resolution.

Feature requests: what underlying problem are you trying to solve with this feature?

Be able to build on old versions of macOS. Unless minimum version of macOS is planned and documented somewhere?

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

• get a macbook with 10.13.6, and XCode CLT installed.
• checkout bazelbuild/bazel at HEAD today
• install JDK, e.g., via brew tap AdoptOpenJDK/openjdk ; brew install adoptopenjdk11
• bazel build //src:bazel_nojdk with 4.0.0

Try again with 3.7.2, and it builds successfully.

What operating system are you running Bazel on?

macOS High Sierra 10.13.6 or Mojave 10.14.2 or Mojave 10.14.6

What's the output of bazel info release?

release 3.7.2 or release 4.0.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

Redundant with above, but built from 660f5b2

Have you found anything relevant by searching the web?

No. I specifically looked for release notes announcing this as a breaking change, didn't find any here or here

[dhalperi]

Since High Sierra is effectively EOL now (no longer receiving security patches as of this month), probably skip worrying about that one.

I have obtained a test machine and can repro with Mojave easily.

[keith]

Can you verify the error you're seeing in the latest supported Xcode version on that OS is the same as the libtool one above and not #12905 ?

If it is the libtool issue it's likely #12265 in which case we'd likely need to add some OS version checking to disable this behavior

[dhalperi]

I just expunged and removed /var/tmp/_bazel_$USER, and made sure I was on 4.0.0.

I am not 100% certain I have the "latest supported Xcode version" -- Apple has made this harder to get recently.

Command Line Tools 11.3 requires 10.15.4 minimum, so won't work on Mojave.

I think I have CLT 10.3 ish:

xcode-select -v gives me version 2354.
clang -v gives me Apple LLVM version 10.0.1 (clang-1001.0.46.4)
==> Running pkgutil --pkg-info=com.apple.pkg.CLTools_Executables gives me version 10.3.0.0.1.1562985497

This is the latest returned by xcode-select --install (which I ran immediately after installing Mojave). After that it tells me to use Software Update, which has no updates for me.

So that's the caveat, but with that, yes: the error I get is the same libtool error above.

[keith]

The version question here is just what version of Xcode added support for params files to libtool. It looks like the answer to that is 11.0

[dhalperi]

Update!

I went back to the developer tools site, found that I was wrong about minimum version requirements. 11.5 requires 10.15 (Catalina), but 11.3.1 I could install on Mojave. That actually seems to work fine, without any caveats, on Mojave.

version: 11.3.1.0.1.1576735732

[philwo]

Hi, we cannot test Bazel on or develop any patches for macOS Mojave ourselves, as we usually upgrade quickly after a new version comes out. However I'm happy to take patches if someone wants to contribute.

That said, does the last update mean that this issue is resolved now (by upgrading Xcode)?

[dhalperi]

Is it obvious why CI can't have older versions of OS X -- I imagine mainly maintainer time.

I'd say the best resolution would be to advertise expected support for macOS (and other platforms): which versions are tested, which are used by developers, etc. And to update in release notes when they change. Presumably this issue can serve as notification of resolution for macOS users who use Mojave and have developer accounts. (You can't get into the download without one.)

[philwo]

Is it obvious why CI can't have older versions of OS X -- I imagine mainly maintainer time.

Maintainer time and effort is one factor, also limited hardware - we only have 20x iMac Pros to run Bazel CI on and as we don't use virtualization for performance reasons, it's not (easily) possible to run different versions of macOS side-by-side. :/

It's a good idea to document the supported / tested macOS versions, I agree. Currently we are still running Catalina on the CI machines, so all recent Bazel versions up to 4.0 are built on and tested on that version.

[dhalperi]

Makes sense. Thanks @philwo as always - great!

Happy to close this issue if that's preferable.

[xlla]

hello, I run into same issue while install bazel using brew on macOS 10.13, I must stick to this ancient version to use CUDA since Apple and nVidia breakdown each other.

bazel was so complex and it will delete work folder after error raise and I can't trace the params file content.

who can tell me which files involved to generate params file for ar/libtool , I want to add a shim file to emulate new libtool's behaviours to pass the build.

errors with libtool

.WARNING: Option 'host_javabase' is deprecated
INFO: Analyzed target //src:bazel_nojdk (368 packages loaded, 10310 targets configured).
INFO: Found 1 target...
[0 / 28] 8 actions, 0 running
    [Prepa] BazelWorkspaceStatusAction stable-status.txt
    [Prepa] Writing file src/main/cpp/client-2.params
    [Prepa] Writing file src/main/tools/daemonize-2.params
    [Prepa] Writing script external/bazel_tools/tools/cpp/malloc.cppmap
ERROR: /private/tmp/bazel_iUFGQNFN/out/external/bazel_tools/third_party/ijar/BUILD:47:11: Linking external/bazel_tools/third_party/ijar/libzlib_client.a [for host] failed: (Exit 1): libtool failed: error executing command 
  (cd /private/tmp/bazel_iUFGQNFN/out/execroot/io_bazel && \
  exec env - \
    PATH=/usr/local/opt/[email protected]/libexec/bin:/usr/local/Homebrew/Library/Homebrew/shims/mac/super:/usr/local/opt/[email protected]/bin:/usr/local/opt/openjdk@11/bin:/usr/bin:/bin:/usr/sbin:/sbin \
    PWD=/proc/self/cwd \
  /usr/bin/libtool @bazel-out/host/bin/external/bazel_tools/third_party/ijar/libzlib_client.a-2.params)
# Configuration: 14f4112a92717319874531ee87fcebf79151508befc3dc74ce6dc0b97991d1a3
# Execution platform: //:default_host_platform
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: no output file specified (specify with -o output)
Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -static [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-sacLT] [-no_warning_for_no_symbols]
Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-o output] [-install_name name] [-compatibility_version #] [-current_version #] [-seg1addr 0x#] [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table ] [-seg_addr_table_filename ] [-all_load] [-noall_load]
Target //src:bazel_nojdk failed to build
INFO: Elapsed time: 28.645s, Critical Path: 0.44s
INFO: 47 processes: 42 internal, 5 local.
FAILED: Build did NOT complete successfully

ERROR: Could not build Bazel

errors with ar

🍃  Building Bazel from scratch......
🍃  Building Bazel with Bazel.
.WARNING: Option 'host_javabase' is deprecated
INFO: Analyzed target //src:bazel_nojdk (368 packages loaded, 10310 targets configured).
INFO: Found 1 target...
[0 / 30] 6 actions, 0 running
    [Prepa] BazelWorkspaceStatusAction stable-status.txt
    [Prepa] Writing file src/main/cpp/client-2.params
    [Prepa] Writing file src/main/tools/linux-sandbox-2.params
    [Prepa] Writing file src/embedded_tools_nojdk.params
ERROR: /private/tmp/bazel_pe86hMz0/out/external/bazel_tools/src/main/cpp/util/BUILD:34:11: Linking external/bazel_tools/src/main/cpp/util/libfilesystem.a [for host] failed: (Exit 1): ar failed: error executing command 
  (cd /private/tmp/bazel_pe86hMz0/out/execroot/io_bazel && \
  exec env - \
    PATH=/usr/local/opt/[email protected]/libexec/bin:/usr/local/Homebrew/Library/Homebrew/shims/mac/super:/usr/local/opt/[email protected]/bin:/usr/local/opt/openjdk@11/bin:/usr/bin:/bin:/usr/sbin:/sbin \
    PWD=/proc/self/cwd \
  /usr/bin/ar @bazel-out/host/bin/external/bazel_tools/src/main/cpp/util/libfilesystem.a-2.params)
# Configuration: 14f4112a92717319874531ee87fcebf79151508befc3dc74ce6dc0b97991d1a3
# Execution platform: //:default_host_platform
usage:  ar -d [-TLsv] archive file ...
	ar -m [-TLsv] archive file ...
	ar -m [-abiTLsv] position archive file ...
	ar -p [-TLsv] archive [file ...]
	ar -q [-cTLsv] archive file ...
	ar -r [-cuTLsv] archive file ...
	ar -r [-abciuTLsv] position archive file ...
	ar -t [-TLsv] archive [file ...]
	ar -x [-ouTLsv] archive [file ...]
Target //src:bazel_nojdk failed to build
INFO: Elapsed time: 29.343s, Critical Path: 1.23s
INFO: 87 processes: 67 internal, 20 local.
FAILED: Build did NOT complete successfully

ERROR: Could not build Bazel

Can I build bazel use other compiler like llvm/gcc instead of xcode to avoid this problem?

thanks for any suggestion.

[dhalperi]

At this point 4 is one major version behind, and Mojave is 3 behind (soon to be 4), I think it's worth closing.