bazel_4: init at 4.0.0 #106984
bazel_4: init at 4.0.0 #106984
Conversation
|
Awesome PR! Good to have early builds of new bazel to test scala 3.0 without custom nixpkgs overrides. |
|
Bazel 4.0 just has been released... 🎉 https://github.com/bazelbuild/bazel/releases/tag/4.0.0 Looking forward for an update here. 😄 (BTW, does this PR build on darwin?) |
|
Updated to 4.0.0: squash, rebase and new title&description |
|
Works for me on linux, but my colleague using MacOS big sur got the following error when trying to build: Had a quick look around and found somebody suggesting to pass |
|
Interesting, there seem to be some patches to use clang from nixpkgs |
|
I've added |
|
Any clues @divanorama ? |
|
No, I don't know how to test on darwin, is there a public VM image or something? |
|
By the way, would Hydra / NixPkgs infrastructure compile it on macos or cross-compile on matching arch with Nix? Maybe we're lucky and it'd populate nix cache? |
There are Hydra builds for macos, so that is fine. From the checks that ran in this PR it appears that the tests are failing on Linux, so I don't think it would be merged before the Linux tests are passing. Those need to be working first before any Darwin issues are investigated I think. I think the aarch64-linux tests aren't expected to pass for this package. Only linux and darwin. |
|
Didn't notice the linux breakage, thanks! Will try to reproduce locally |
|
checks are green now 🥳 |
Awesome! I've quickly tried locally on Catalina (10.15.7) and it fails as follows: |
|
thanks, interesting bit is so it's likely that clang 6 from NixPkgs is used, but fails Wonder if diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index bea1931aac9..b8e818288aa 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -11673,7 +11673,6 @@ in
buildJdk = jdk8_headless;
buildJdkName = "jdk8";
runJdk = jdk11_headless;
- stdenv = if stdenv.cc.isClang then llvmPackages_6.stdenv else stdenv;
bazel_self = bazel_4;
};or maybe diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index bea1931aac9..4485534bc52 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -11673,7 +11673,7 @@ in
buildJdk = jdk8_headless;
buildJdkName = "jdk8";
runJdk = jdk11_headless;
- stdenv = if stdenv.cc.isClang then llvmPackages_6.stdenv else stdenv;
+ stdenv = if stdenv.cc.isClang then llvmPackages_7.stdenv else stdenv;
bazel_self = bazel_4;
};can help |
|
I'll try those. It definitely seems like the error is similar to this suggestion from @kalbasit and @avdv #105439 (comment) It seems like the linker is being passed a strange version of
|
|
Just for my own sanity, I've confirmed that it is possible to build Bazel 4.0.0 from source on my laptop running macos 10.15.7. That at least gives me some comfort that it should be possible to get it building as a nixpkg. |
|
@divanorama In terms of clang-7 on macos, it doesn't appear to work.
I'll need to pause my investigations for now. |
I also thought about upgrading clang and ran into the same issue. I found out that libarclite is the automatic reference counting library of objective C which is needed when you use it within your code. Which boils down to the question why clang 6 is able to compile this code but clang 7 fails. Could this be an issue within clang 7 itself?! Where does this library come from? |
|
https://discourse.nixos.org/t/linker-file-not-found-lib-arc-libarclite-macosx-a-on-darwin/10486 Which could be coming from https://github.com/bazelbuild/bazel/blob/4.0.0/scripts/bootstrap/compile.sh#L410 Though it was there in bazel-3.3.1 too, maybe clang-6 in NixPkgs can pick it up from host but clang-7 doesn't? |
|
Interestingly, this seems to improve things. diff --git a/pkgs/build-support/bintools-wrapper/default.nix b/pkgs/build-support/bintools-wrapper/default.nix
index 48a3ebb32df..49d662b263f 100644
--- a/pkgs/build-support/bintools-wrapper/default.nix
+++ b/pkgs/build-support/bintools-wrapper/default.nix
@@ -252,7 +252,7 @@ stdenv.mkDerivation {
# Ensure consistent LC_VERSION_MIN_MACOSX and remove LC_UUID.
+ optionalString stdenv.targetPlatform.isMacOS ''
- echo "-sdk_version 10.12 -no_uuid" >> $out/nix-support/libc-ldflags-before
+ echo "-macosx_version_min 10.12 -sdk_version 10.12 -no_uuid" >> $out/nix-support/libc-ldflags-before
''
##This is similar advice to the comment here from @kalbasit: #105439 (comment) It takes too long to build with this patch, so I'm not sure how to set The error then progresses to: |
|
I've been trying to work out if the versoin of It seems that the thread_local was introduced into I've yet to work out the dependency graph that pulls in I'll have to pause my investigations for now. Summary of my view of the situation on Darwin
|
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/nixpkgs-bazel-4-darwin-assistance/11303/1 |
|
It appears that I believe the dependencies are as follows: It appears that Bazel 3.3.1 (release on 1 July 2020) had no directly listed dependency on From what I can gather: |
|
re2 had some fixes to disable thread local google/re2@f2a896b and google/re2@d8d0c66 Not clear whether clang on darwin in nix should have thread local support or not though. |
I believe that macro is to disable it for devices such as iphone, ipad etc. The authors would expect a mac to have thread_local support because it isn't written with Nix in mind, it is written expecting a standard OSX user with XCode etc. That is my reading of it anyway.
Yes, and if nixpkgs clang on darwin does not support thread_local, packages such as I'm not entirely sure where this leaves us. I think we either need some nixpkgs Darwin experts or it might be the end of the road for nixpkgs Bazel on Darwin? I'm not sure how feasible it would be to patch ourselves out of this problem very easily. From my perspective I would still prefer to have bazel available on darwin nixpkgs. |
|
@LnL7 do you have any idea if nixpkgs could support clang thread_local and how much work could that take? |
|
As for patching it, this https://github.com/divanorama/nixpkgs/pull/1/files may do it. #if defined(__APPLE__)
#undef RE2_HAVE_THREAD_LOCAL
#endifThere may be other failures, including on thread local, further down the build though |
🎉 That has got it working locally for me @divanorama (with one large caveat) Caveat: Can you ask your friend with Big Sur to see if it works without the patch? So I think we are close, but I still think this |
|
In breaking news, @avdv may have found a patch to get things working on Clang7 which avoids the |
Bazel 4 is going to be a long term support release. Latest version in NixPkgs so far was 3.3.1 There's a need for more recent version NixOS#97497 All versions from 3.5.0 to 3.7.1 had some reproducibility issues as noted in issue above, but there also seems to be a working PR for 3.7.1 now at NixOS#105439 Notable changes from bazel_3 setup: - put python to default bash path For autodetecting python toolchain with strict action_env on and without this change bazel would fail to autodetect host python. There are some repos that define hermetic python toolchains, but they aren't easy to use yet. Also telling python paths to bazel isn't a 1-liner it seems: - action_env=PATH would affect cache - declaring toolchain via BUILD&WORKSPACE files is not per-user but more like per-repo and affects cache too Using python from nixpkgs shouldn't be too bad in the lack of simpler hermetic python toolchain options - bazel_4.updater is bazel on `bazel query` to support new constructs in WORKSPACE (load of vars, transitive load etc). This is more robust but requires bazel to run the updater, using bazel_3 for now. This is only needed to bump package version, doesn't introduce bazel_4 build dependency on bazel_3 https://blog.bazel.build/2020/11/10/bazel-4.0-announce.html https://blog.bazel.build/2020/11/10/long-term-support-release.html bazelbuild/bazel#12455 https://github.com/bazelbuild/bazel/releases/tag/4.0.0 https://blog.bazel.build/2021/01/19/bazel-4-0.html
- bump protobuf to 3.13.0 - use recent rules_proto - use recent bazel-skylib - remove use of native.bind Hopefully this is backward-compatible with bazel_3 and below
* use default stdenv (clang 7) * add no-arc.patch to make the xcode_locate tool compile without libarc-lite * remove the `-mmacosx-version-min=10.9` flag from the bootstrap compile script
Use the same JDK for building bazel and for its runtime.
Effectively, the former `toolchain_hostjdk8` java toolchain has been deprecated
and should no longer be used (in newer bazel)[1]:
```
default_java_toolchain(
name = "toolchain_hostjdk8",
...
)
```
[1]: https://github.com/bazelbuild/bazel/blob/4fc48680653a71aacbfd555436ba8f9a0742d3d9/tools/jdk/BUILD.tools#L384-L387
|
Thanks a lot @divanorama 🚀 Apologies for sending you down the rabbit hole to patch |
|
This works for me locally on Darwin. 🎉 Is there something more to be done before this can be merged @mboes @Profpatsch ? |
|
@Profpatsch I don't have merge rights so passing the puck to you. |
|
I can merge it.. |
Bazel 4 is going to be a long term support release.
Latest version in NixPkgs so far was 3.3.1
There's a need for more recent version
#97497
All versions from 3.5.0 to 3.7.1 had some reproducibility issues
as noted in issue above, but there also seems to be
a working PR for 3.7.1 now at
#105439
Notable changes from bazel_3 setup:
put python to default bash path
For autodetecting python toolchain
with strict action_env on and without this change
bazel would fail to autodetect host python.
There are some repos that define hermetic python
toolchains, but they aren't easy to use yet. Also
telling python paths to bazel isn't a 1-liner it
seems:
is not per-user but more like per-repo and
affects cache too
Using python from nixpkgs shouldn't be too bad
in the lack of simpler hermetic python toolchain
options
bazel_4.updater is bazel on
bazel queryto supportnew constructs in WORKSPACE (load of vars, transitive
load etc). This is more robust but requires bazel
to run the updater, using bazel_3 for now. This is
only needed to bump package version, doesn't introduce
bazel_4 build dependency on bazel_3
https://blog.bazel.build/2020/11/10/bazel-4.0-announce.html
https://blog.bazel.build/2020/11/10/long-term-support-release.html
bazelbuild/bazel#12455
https://github.com/bazelbuild/bazel/releases/tag/4.0.0
https://blog.bazel.build/2021/01/19/bazel-4-0.html