Multiple downstream failures due to --incompatible_use_platforms_repo_for_constraints

[Wyverald]

Failures: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2586#0182a995-9a6f-49fc-a38d-06487e43e242

Flag flip commit: bazelbuild/bazel@3469784

Autosheriff link: https://buildkite.com/bazel/bazel-auto-sheriff-face-with-cowboy-hat/builds/1005

Projects to fix:

• Bazel
• Bazel integration testing: removed Remove bazel-integration-testing from downstream pipeline #1405
• Bazelisk: sent update rules_nodejs version bazelisk#348
• Buildfarm: sent update rules_k8s version buildfarm/buildfarm#1142 (no response; disabled in disable Buildfarm and Cartographer in downstream #1416)
• Cartographer: sent update rules_boost to latest version cartographer-project/cartographer#1898 (no response; disabled in disable Buildfarm and Cartographer in downstream #1416)
• Envoy: they unflipped the flag for now; ongoing work tracked at Please regenerate RBE toolchain configs envoyproxy/envoy#22758
• Flatbuffers: sent update grpc version google/flatbuffers#7457
• Kythe: sent update googletest version kythe/kythe#5358
• Protobuf: sent update bazel_skylib to latest version (1.2.1) protocolbuffers/protobuf#10423
• TensorFlow: filed Bazel CI Failing for Tensorflow with Bazel@HEAD tensorflow/tensorflow#57207
• rules_cc: sent Update WORKSPACE to account for bazel_tool//platforms going away. rules_cc#142
• rules_foreign_cc
• rules_go
• rules_groovy
• rules_jvm_external
• rules_jvm_external - examples
• rules_kotlin
• rules_nodejs
• rules_proto: @aranguyen sent migrating to usage of rbe_preconfig and remove bazel-toolchains rules_proto#134
• rules_rust
• rules_sass: sent update @io_bazel_rules_go//go:def.bzl for https://github.com/bazelbui… rules_sass#143
• upb: turns out upb main build last-green has been stuck ever since they switched from master to main. I just re-pointed it at main, and sent install rsync on Bazel CI before running tests protocolbuffers/upb#738 to fix the main build. After last-green is updated, downstream should automatically turn green.

[aiuto]

Interesting. The configurability team will have to look into these. Even if we were to roll it back, we still want the change for Bazel 6.x, so fixing the rules is the right choice.

…

On Wed, Aug 17, 2022 at 10:16 AM Xùdōng Yáng ***@***.***> wrote: Failures: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2586#0182a995-9a6f-49fc-a38d-06487e43e242 Flag flip commit: ***@***.*** <bazelbuild/bazel@3469784> Autosheriff link: https://buildkite.com/bazel/bazel-auto-sheriff-face-with-cowboy-hat/builds/1005 Projects to fix: - Bazel - Bazel integration testing - Bazelisk - Buildfarm - Cartographer - Envoy - Flatbuffers - Kythe - Protobuf: sent protocolbuffers/protobuf#10423 <protocolbuffers/protobuf#10423> - TensorFlow - rules_cc - rules_foreign_cc - rules_go - rules_groovy - rules_jvm_external - rules_jvm_external - examples - rules_kotlin - rules_nodejs - rules_proto - rules_python - rules_rust - rules_sass - upb — Reply to this email directly, view it on GitHub <#1404>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHBNYN6VYDGVXJIKEKDVZTX5RANCNFSM56ZXGBVA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[aranguyen]

@Wyverald A good number of the rest of the projects are caused by the usage of older bazel-toolchains which is linked to RBE. bazelbuild/rules_proto#134 is an example of removing deprecated rbe_autoconfig and migrating to rbe_preconfig. In the same PR, I also removed bazel-toolchains because there is no other usage. If there are, then updating it to 5.1.2 & use rbe_preconfig is the correct thing to do. I need to divert my attention to something else for the next few hours so I just want to share this in case you get to them before me. I will be back. Thanks!

[Wyverald]

thanks for the update!

[aranguyen]

Just another update, I've sent prs for the rest except for rules_kotlin
rules_nodejs, rules_cc and TensorFlow. @Wyverald is there anything on this list that you're actively working on?

[Wyverald]

Thanks, Ara. I've been busy with other stuff so haven't been sending new PRs for anything else.

For the other PRs you sent, could you mention this issue in them so I can keep track?

[aranguyen]

Okay sounds good. I will make sometime today to work on the rest then. I sent these

1. rules_go migrating to rbe_preconfig bazel-contrib/rules_go#3272
2. rules_foreign_cc migrating to rbe_preconfig bazel-contrib/rules_foreign_cc#952
3. rules_rust migrating to rbe_preconfig and remove bazel_toolchains rules_rust#1524
4. rules_proto migrating to usage of rbe_preconfig and remove bazel-toolchains rules_proto#134
5. rules_groovy migrating to rbe_preconfig rules_groovy#62

The above were all merged. The followings are still open
6. rules_jvm/rules_jvm_external bazel-contrib/rules_jvm_external#730
7. rules_sass bazelbuild/rules_sass#143
8. rules_nodejs bazel-contrib/rules_nodejs#3541

[alexeagle]

@aranguyen the commit in rules_nodejs didn't actually enable the flag https://github.com/bazelbuild/rules_nodejs/pull/2536/files#diff-544556920c45b42cbfe40159b082ce8af6bd929e492d076769226265f215832fR56

I'll try again now...

[aranguyen]

@alexeagle that comment I made on your pr previously is a bit ancient. I think it was during the time when I updated the external deps for bazel so that is obsolete. The failure for rules_nodejs we're seeing now should be addressed by this pr bazel-contrib/rules_nodejs#3541 . Similar reason mentioned here #1404 (comment)

[aranguyen]

Update:

1. rules_proto I sent an additional pr update googletest rules_proto#137
2. rules_kotlin has these remaining references

aranguyen-macbookpro:rules_kotlin aranguyen$ grep -r "@bazel_tools//platforms" /private/var/tmp/_bazel_aranguyen/296329cf59a3bbb8fa169f1cac45cd9a
/private/var/tmp/_bazel_aranguyen/296329cf59a3bbb8fa169f1cac45cd9a/external/rules_jvm_external/examples/android_instrumentation_test/BUILD:        "@bazel_tools//platforms:x86_64",
/private/var/tmp/_bazel_aranguyen/296329cf59a3bbb8fa169f1cac45cd9a/external/rules_jvm_external/examples/android_instrumentation_test/BUILD:        "@bazel_tools//platforms:linux",

I need to wait for this PR in rules_jvm_external to be reviewed and merged first before I can update rules_kotlin
3) rules_cc : I am not able to build it on my machine for another reason so I filed this issue instead bazelbuild/rules_cc#140 . It has my analysis and what's needed to be done. I will try again tomorrow. Hopefully I receive some guidance on the build issue and I can help with the update as well.

I also saw an internal cl for Tensorflow so at this point all failed downstream deps have been looked at and actions were taken. Please let me know if I'm missing anything.

[mai93]

I updated bazel_skylib version in rules_jvm_external bazel-contrib/rules_jvm_external#742 but some of tests are still failing with this error

(02:23:26) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/627400a4322f538dd0c1a564239629b9/external/bazel_tools/third_party/ijar/BUILD:47:11: error loading package '@bazel_tools//third_party/zlib': Unable to find package for @rules_license//rules:license.bzl: The repository '@rules_license' could not be resolved: Repository '@rules_license' is not defined. and referenced by '@bazel_tools//third_party/ijar:zlib_client'
 
Does anyone have an idea what needs to be updated to fix it?

[Wyverald]

I have a pending CL for that.

[Wyverald]

As of today, the following projects remain broken due to this issue (downstream pipeline link):

• FlatBuffers
• TensorFlow
• rules_cc
• rules_jvm_external (RBE only)
• rules_jvm_external - examples
• rules_kotlin (example - node)
• rules_sass
• upb

The other breakages in downstream are unrelated and need to be fixed separately.

[aranguyen]

@Wyverald thanks for letting me know. I'll take a look at rules_cc, jvm_external, and rules_kotlin again to see why the prs I sent out were not sufficient enough. For rules_sass, we have a pending pr that has been sitting there for a long time unreviewed bazelbuild/rules_sass#143

it possible to do a clean --expunge beforehand when running the downstream pipeline? I remember there was a discussion about how rbe_preconfig is fetching something in a non-deterministic way?

[Wyverald]

I believe we effectively already do clean --expunge (we start a new docker instance for each job). rbe_preconfig is special in that it fetches from a URL whose contents change; I don't remember how it affected CI.

[Wyverald]

Worth noting, rules_go (on all platforms except RBE) also seems to be broken due to this change, even though the error message doesn't look the same: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2618#01831d43-1066-44ad-9289-f8b72b54f294

In a downstream run with the flag unflipped, rules_go is fine: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2620#01831d5e-da23-40c7-84fe-9b9d5a6244ee

[meteorcloudy]

@aranguyen Given this flag flipped has caused our downstream pipeline to be red for a long time, do you mind we first rollback the flag flip? You can still monitor the downstream projects status in https://buildkite.com/bazel/bazelisk-plus-incompatible-flags after unflip the flag. This will help us keep downstream green and catch actual Bazel bugs.

[aranguyen]

@meteorcloudy we actually want the flag flip for the 6.0 release. If we rollback the flag flip, when would be the latest we can roll forward the change for 6.0?

[Wyverald]

flatbuffers: upgrading grpc to 1.48.0 wasn't enough; we need a version of grpc that contains grpc/grpc@27509c3, and as of today there's no stable release containing it yet

[aranguyen]

I submit an internal cl to flip the default value of --incompatible_use_platforms_repo_for_constraints to false for now. Two more prs are sent out:

• rules_jvm_external update bazel skylib and use rbe_preconfig bazel-contrib/rules_jvm_external#746
• rules_cc update to latest version of rules_go rules_cc#144