Bazel 0.5.0 build fails on Mac OS X El Capitan

[zanes2016]

Description of the problem / feature request / question:

Bazel build fails on Mac OS X El Capitan (Version 10.11.6). The same build completes without any issue on Ubuntu 14.04. It looks like this is a mac specific issue.

Mac (build error):

$ bazel version
Build label: 0.5.0-homebrew
Build target: bazel-out/darwin_x86_64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri May 26 14:50:47 2017 (1495810247)
Build timestamp: 1495810247
Build timestamp as int: 1495810247
$
$
$ sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.11.6
BuildVersion:	15G1510
$
$
$ bazel build --verbose_failures :all
ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': Traceback (most recent call last):
	File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 829
		_get_escaped_darwin_cxx_inc_directories(repository_ctx, cc)
	File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 726, in _get_escaped_darwin_cxx_inc_directories
		for toolchain in toolchains: ...
type 'NoneType' is not iterable.
INFO: Elapsed time: 3.445s

Ubuntu (no error):

$ bazel version
Build label: 0.5.0
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri May 26 12:11:50 2017 (1495800710)
Build timestamp: 1495800710
Build timestamp as int: 1495800710
$
$
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.4 LTS
Release:	14.04
Codename:	trusty
$
$
$ # I already ran the build, it succeeded without any issues
$ bazel build --verbose_failures :all
INFO: Found 1 target...
Target //:env up-to-date (nothing to build)
INFO: Elapsed time: 0.128s, Critical Path: 0.00s

If possible, provide a minimal example to reproduce the problem:

On a mac, the following produces the error

$ bazel build --verbose_failures :all

Environment info

• Operating System:
  ProductName: Mac OS X
  ProductVersion: 10.11.6
  BuildVersion: 15G1510

• Bazel version (output of bazel info release):
  release 0.5.0-homebrew

Anything else, information or logs or outputs that would be helpful?

I'm trying to build the same project on both mac and ubuntu which is version controlled with git. So i'm confident that the two build sources are identical.

[damienmg]

Can you try to rebuild after bazel clean --expunge?

[damienmg]

(I use macOS El Capitan without issue, btw)

[zanes2016]

I still have the same issue:

$ bazel clean --expunge
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
$
$ bazel build :all
..............................................
ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': Traceback (most recent call last):
	File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 829
		_get_escaped_darwin_cxx_inc_directories(repository_ctx, cc)
	File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 726, in _get_escaped_darwin_cxx_inc_directories
		for toolchain in toolchains: ...
type 'NoneType' is not iterable.
INFO: Elapsed time: 4.959s

[damienmg]

/cc @mhlopko undetected regression of 0.5.0 :(

@c-parsons this is happening because xcode_locator does not returns any toolchain, I don't know how it works but if we do not find a user side workaround, we need to do a patch release.

[damienmg]

@zanes2016: as much information about your xcode installation would help. gcc --version also

[zanes2016]

$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[damienmg]

/cc @c-parsons now Bazel is failing to build C++ code if you don't have xcode installed.

@zanes2016 seems like my only fix right now is to install xcode :( sorry for that.

[c-parsons]

I've thrown together a fix for this, sending it to you right now, Damien.
Sorry for the breakage -- we apparently don't have a good CI environment to exercise this situation.

[damienmg]

Well if installing xcode fix the issue this might be an acceptable workaround. 0.5.1 should be in less than a month.

Yes having all the use case in a CI is hard.

[MarkusTeufelberger]

Looks like #3060 is another instance of this?

[damienmg]

Indeed it is, we will start seeing a lot of bug report, good thing I am in the US right now, I'll might cut a patch release if that start to have a lot of feedback.

[damienmg]

(but first I would like to confirm @c-parsons fix is working, which I believe should be ok)

[damienmg]

Any idea how to test it?
@zanes2016 or @zhexuany, would any of you willing to help in an interactive session (talking on IRC or hangout)?

[zanes2016]

@damienmg I should be available for a hangout in about 20 minutes

[zhexuany]

@damienmg I am available right now.

[damienmg]

Best plan of action we have right now:

1. Test @c-parsons change on @zanes2016 or @zhexuany machine (or anyone hitting the issue)
2. Roll-back update to 0.5.0 for bazel (bazel 0.5.0 Homebrew/homebrew-core#13967, /cc @ilovezfs)
3. Create a 0.5.1rc1 that just include @c-parsons change today
4. Have 0.5.1rc1 released early next week, we would have to have an exception to our release policy.
5. Bump to 0.5.1 for the release in homebrew.

Up to 4 there is no need for further agreement, I let the rest of the Munich team decide on 3 on Monday while the US are on holidays.

@zanes2016 @zhexuany thanks! mail me at [email protected] so we can chat on hangout (or ping me on irc freenode / dmarting). I need 10-20 min to prepare the rc branch

[zanes2016]

For completeness, I've also verified that this issue exists on MacOS Sierra:

$ sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.12.5
BuildVersion:	16F73
$
$
$ bazel build :all
ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': Traceback (most recent call last):
	File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 829
		_get_escaped_darwin_cxx_inc_directories(repository_ctx, cc)
	File "/private/var/tmp/_bazel_user/6f6133ece88806ce1513398e26791cf1/external/bazel_tools/tools/cpp/cc_configure.bzl", line 726, in _get_escaped_darwin_cxx_inc_directories
		for toolchain in toolchains: ...
type 'NoneType' is not iterable.
INFO: Elapsed time: 0.203s

[damienmg]

@zhexuany tested it with Xcode, still failing, might need more than the default install. With definitely needs a patch release.

[zhexuany]

ERROR: /private/var/tmp/_bazel_zhexuany/6bfb8836d8a7c642713005d68a02df63/external/local_config_cc/BUILD:57:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-darwin_x86_64: Xcode version must be specified to use an Apple CROSSTOOL.
ERROR: Analysis of target '//src/test/java/com/pingcap/tikv:AllTests' failed; build aborted.
INFO: Elapsed time: 33.385s
ERROR: Couldn't start the build. Unable to run tests.

hope this can help in some senses.

[damienmg]

FTR the test rc failed to build the release notes, I'll can extract the binary for test but a patch release seems like it has a hard time or it is because I tweaked the cherry-picking.

[damienmg]

The darwin package has been suceesfully built so we can at least test the fix itself.

[hlopko]

I fully agree, let's wait for @c-parsons to tell us what parts of Xcode Bazel needs.

[damienmg]

US are on holiday today. The principle fix will takes several patch to do, I was leaning toward a rollback but the rollback has conflict with 2 of your changes, it was already late so I just handover the bug to you. Homebrew has reverted the update so no new client will install it, avoiding more people coming here. I added a message to the release notes page, maybe it is not really visible. Bazel does not requires Xcode to build C++ code, only to bootstrap. We should keep it that way.

…

On Mon, May 29, 2017 at 4:32 AM Marcel Hlopko ***@***.***> wrote: I fully agree, let's wait for @c-parsons <https://github.com/c-parsons> to tell us what parts of Xcode Bazel needs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3063 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADjHf9yyLElnoXeI0UmHiiETD9vqxi0yks5r-qzhgaJpZM4NoEvI> .

[c-parsons]

Hello folks.
Indeed rolling back the apple crosstool changes would be a difficult task given the number of dependent changes that are in the release.
I will quickly try to draft a patch change which will turn off apple crosstool if xcode is not present.

Sorry for the inconvenience.

[hlopko]

Hello everybody,
Chris was super fast with the fix, and we have a 0.5.1RC1 ready. Would you mind testing it if it fixes the problem for you? You can get it from

https://storage.googleapis.com/bazel/0.5.1/rc1/index.html

If yes, I'll close this issue. Thanks!

[ittaiz]

Marcel, 0.5.1 will only contain this fix, right?

…

On Wed, 31 May 2017 at 18:17 Marcel Hlopko ***@***.***> wrote: Hello everybody, Chris was super fast with the fix, and we have a 0.5.1RC1 ready. Would you mind testing it if it fixes the problem for you? You can get it from https://storage.googleapis.com/bazel/0.5.1/rc1/index.html If yes, I'll close this issue. Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3063 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF-v-in6R55kyWhFB9grTJ6iJ4wUUks5r_YSPgaJpZM4NoEvI> .

[evanj]

0.5.1RC1 seems to fix the problem for me on OS X 10.11.6. Thanks!

[hlopko]

@ittaiz Yes, only that fix. No new behavior, no need to upgrade bazel if you don't need the fix.
@evanj Awesome! I'm letting it sit for 2 days and then I'll release it and upgrade the homebrew recipe. Thank you all!

[ilovezfs]

I've now shipped 0.5.1 in Homebrew.

[zhexuany]

@c-parsons
Still failing.
ERROR: /private/var/tmp/_bazel_zhexuany/d90cfb1d83b5fac0d2a4fea11702e03d/external/local_config_cc/BUILD:57:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-darwin_x86_64: Xcode version must be specified to use an Apple CROSSTOOL.

[hlopko]

Just to be sure, also after bazel clean --expunge?

[zhexuany]

My bad. Now error is gone. Thanks.

[philipposslicher]

Error is still present in 10.13 High Sierra

[varming]

Ran into this error today after upgrading to 0.5.2 in homebrew.

[damienmg]

@philipposslicher @varming: reopening a new bug.

[phynero]

ERROR: /private/var/tmp/_bazel_phynero/759a3c362e95012bc330dc146849fc13/external/local_config_cc/BUILD:49:5: in apple_cc_toolchain rule @local_config_cc//:cc-compiler-darwin_x86_64: Xcode version must be specified to use an Apple CROSSTOOL.
ERROR: Analysis of target '//:add_person_java_lite' failed; build aborted.

T-T WHAT CAN IDO

[phynero]

xcode_config(name = 'host_xcodes')

Error: Running xcodebuild -version failed, return code 72, stderr: xcrun: error: unable to find utility "xcodebuild", not a developer tool or in PATH , stdout:

[c-parsons]

What type of target are you trying to build?

[nornagon]

I fixed this by running sudo xcodebuild -license and agreeing to the TOS.

(Sadly, I then encountered clang: error: unknown argument: '-fno-canonical-system-headers')

[dpursehouse]

I just had this problem on OSX 10.12.6 with bazel 0.5.4, and it was fixed by running bazel clean --expunge

[c-parsons]

dpursehouse: Can you provide more information? Did you have XCode installed beforehand? What kind of project were you trying to build?

[dpursehouse]

@c-parsons yes, I have XCode (version 9.0) installed. I was building the gerrit project.

The error appeared suddenly, having not appeared before. I had not upgraded bazel or XCode.

The only thing that might be related is that I regularly switch between and build different branches.