Release 0.21 - December 2018 #6493

laurentlb · 2018-10-24T11:15:05Z

Target RC date: December 3rd

laurentlb · 2018-12-03T19:04:25Z

Candidate created with:
$ scripts/release/release.sh create --force_rc=2 0.21.0 cb9b2afbba3f8d3a1db8bf68e65d06f1b36902f5
https://releases.bazel.build/0.21.0/rc2/index.html

However some tests are failing: #6832

davido · 2018-12-03T20:56:14Z

Gerrit Code Review cannot be built with this RC, because of outdated protobuf version in rules_closure. Is this expected, to flip the default for any option, until some core bazel rules still depend on those options?

This is the issue in Gerrit issue tracker.
This is the PR for the rules_closure.
This is the fork of rules_closure with this PR included.
This is the CL, that switched to the forked rules_clsoure for Gerrit project.

jin · 2018-12-03T23:20:03Z

@davido I think everyone's waiting for 3.6.1.2 to be released protocolbuffers/protobuf#4650 (comment)

davido · 2018-12-04T05:31:27Z

@jin Thanks for the pointer, but why Bazel is flipping the bit before 3.6.1.2 was released, and before the update to consume protobuf from HEAD is merged? (Not to mention, that even when protobuf 3.6.1.2 is released, reviews for PRs for rules_closure project take ages.)

IOW: until core Bazel rules still depend on deprecated options, Bazel must not flip the bit.

"Nothing causes more pain, frustration, and disappointment than unfulfilled expectations."

I upgrade to Bazel: 0.21.0rc2, and I cannot build my project any more, and even worse: I cannot fix it either.

jin · 2018-12-04T06:29:24Z

@davido I agree, and this has caused me a fair bit of trouble with other projects, which was only really alleviated by passing the incompatible flag.

@meteorcloudy I remember reading somewhere that there’ll be a CI pipeline that tests with all_incompatible_changes. Is that the case, and will that catch a breakage like this? It feels like a really bad user experience to flip a default when upstream dependencies haven’t even remotely caught up yet.

cc @dslomov

davido · 2018-12-04T07:33:57Z

It feels like a really bad user experience to flip a default when upstream dependencies haven’t even remotely caught up yet.

Right, so can we avoid destroying other people work? I reverted 86013a0 here: [1]. Can this CL be merged and cherry-picked on 0.21 branch?

[1] https://bazel-review.googlesource.com/c/bazel/+/83630

laurentlb · 2018-12-04T10:54:53Z

Hi,
We test a number of downstream projects on the CI. Protobuf was updated 6 months ago, other rules were updated more recently. The only exception is rules_closure, but there are pending PRs (it's one-click from being fixed). I think Gerrit just need to update its dependencies too.

I agree that the slow releases of Protobuf are annoying, which is why we want to separate the repository (protocolbuffers/protobuf#5402).

What else shall we update?

meteorcloudy · 2018-12-04T12:03:18Z

We now have a new pipeline for testing Bazel@Release + incompatible flags + downstream projects@last_green_commit
See this document for more details

It will catch those failures earlier and help the downstream projects to migrate.

jin · 2018-12-04T12:55:26Z

This pipeline should help us avoid issues like this in the future. Thanks for working on it! @davido It’ll not be cherrypicked because there’s a workaround by flipping the incompatible_* flags, which is the mechanism to deal with unintended breaking changes.

…

On Dec 4, 2018, at 7:03 AM, Yun Peng ***@***.***> wrote: We now have a new pipeline for testing ***@***.*** + incompatible flags + downstream ***@***.***_green_commit See this document for more details It will catch those failures earlier and help the downstream projects to migrate. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

lberki · 2018-12-05T10:27:32Z

Can we cherry-pick 7fc967c and 798b9a9 ? It solves Android Studio performance issues that arose while migrating to 0.18 (!)

lberki · 2018-12-05T10:27:46Z

/cc @meteorcloudy @dslomov

davido · 2018-12-05T21:41:08Z

@laurentlb

I agree that the slow releases of Protobuf are annoying, which is why we want to separate the repository (protocolbuffers/protobuf#5402).

Protobuf v3.6.1.2 was released today. I've upgraded my PR to bump protobuf to this version in rules_closure.

The only exception is rules_closure, but there are pending PRs (it's one-click from being fixed).

Can someone push merge button then, or what are we missing?

I think Gerrit just need to update its dependencies too.

How can Gerrit update rules_closure, if there is no commit upstream that updated protobuf dependency, and Gerrit maintainers prefer not to fork third party dependencies? In fact my CL that upgraded rules_closure to forked rules_closure version wasn't even reviewed.

@jin

It’ll not be cherrypicked because there’s a workaround by flipping the incompatible_* flags, which is the mechanism to deal with unintended breaking changes.

I disagree. Gerrit Code Review is used by hundreds of core and plugins developers in the wild and we should try very hard to avoid the situation where Gerrit master (and even tip of stable branches) cannot be built with most recent Bazel releases, without passing some obscure --incompatible_foo_bar_baz=false flags.

davido · 2018-12-06T07:53:55Z

So, there is a huge complication with upgrade of protobuf version in rules_closure, unfortunately. I filed this issue.

As pointed out by @acozzette:

We don't have a set date for the 3.7 release, but I'm guessing it should be ready in a month or so.

Why can't we re-consider to revert 86013a0, just merge this CL, and re-visit this deprecation after protobuf 3.7 s released, and Bazel itself, rules_clsoure and Gerrit itself are upgraded to use this full protobuf release?

laurentlb · 2018-12-06T13:55:56Z

@lberki

It solves Android Studio performance issues that arose while migrating to 0.18 (!)

If the same issue is in 0.18, 0.19, and 0.20, it shouldn't be considered as critical for 0.21.

To support building bazel offline just out of the distribution archive, all dependencies that would normally be downloaded during the build are bundled in the distribution archive. So when updating skylib, the @addition_distfiles repository needs to be updated as well. Do so now. To be cherry-picked for #6493, i.e., the bazel 0.21.0 release. Change-Id: I6bf813324298a9b2ca3cb2cac15a871eef9e46f9 PiperOrigin-RevId: 224323158

meteorcloudy · 2018-12-06T14:30:36Z

If the same issue is in 0.18, 0.19, and 0.20, it shouldn't be considered as critical for 0.21.

I don't think I can fully agree. Sounds like as long as a regression exists for a time long enough, it's no longer critical. But there are different reasons for a regression to exist so long:

There is an easy workaround for the regression so that people can endure it.
The project that will hit the regression isn't upgrading Bazel frequently (maybe due to incompatible changes).
The regression is very hard to debug (eg. Android Studio mac performance issues Unbounded thread pool in ByteStreamBuildEventArtifactUploader #6806; TensorFlow malformed output log issue Windows: Malformed Bazel output #6731)

#6806 and #6731 have respectively caused Android Studio and TensorFlow not being able to upgrade Bazel to any version later than 0.18.0. So, I believe they are indeed critical.

Until we can make sure people are actively testing Bazel release candidate (obviously they don't), we should distinguish such regressions.

jin · 2018-12-07T01:46:36Z

@davido I have written a postmortem on how we can prevent this from happening again, using the timeline of the Android projects breakages. I also mentioned the issues surrounding Gerrit and rules_closure, as we both faced similar breakages.

davido · 2018-12-07T06:05:15Z

@jin Thanks, looks good! I commented on this document.

laszlocsomor · 2018-12-07T07:10:44Z

Potential release-blocker: #6859

dslomov · 2018-12-07T09:36:03Z

Sorry for not replying earlier, I was on sick leave.

Re protobuf issue: looks like the world (including Bazel itself, rules_closure and other things) need to move to protobuf 3.6.1.2, and that is an acceptable solution to everyone, is that correct?

Re #6806: this fix should be cherry-picked under a provision for critical, high impact issues. What happened there is that it took as a while to reproduce and investigate this issue, that is why we only got the fix at the date we got it - the issue was reported quite a while ago.

davido · 2018-12-07T10:22:33Z

Re protobuf issue: looks like the world (including Bazel itself, rules_closure and other things) need to move to protobuf 3.6.1.2, and that is an acceptable solution to everyone, is that correct?

Unfortunately, no. Can you see my comments on @jin's postmortem document linked in his previous comment? To summarize:

3.6.1.2 is broken, as it missed at least one commit compared to 3.5.1 protobuf release. That broke rules_closure integrations tests for protobuf_js. I uploaded a PR to fix that. After this PR is merged, new tag 3.6.1.3 must be created. ETA is unknown.
3.6.1.2 is not a complete protobuf release, but just a tag. Particularly protobuf-java wasn't published on Maven Central
Bazel itself still depends on Protobuf 3.6.1
Gerrit Code Review depends on both: rules_closure and on protobuf-java. That why protoc and protobuf-java versions must match, otherwise we are hitting this problem.
Can someone please explain me, why all that mess? Why it is that important to flip the bit: --incompatible_package_name_is_a_function before all that happened:

a) Protobuf 3.7 officially released and all artifacts are published on Maven Central
b) Bazel itself switched to using Protobuf 3.7
c) rules_closure and all other core Bazel rules_xxx projects are adapted to protobuf 3.7 and released

laurentlb · 2018-12-14T16:09:19Z

Cherrypicking 12b9646, 7fc967c, and 798b9a9

aehlig · 2018-12-15T09:02:18Z

rc3 still fails to build offline; can we please cherry-pick dbe05df as requested?

Thanks.

To support building bazel offline just out of the distribution archive, all dependencies that would normally be downloaded during the build are bundled in the distribution archive. So when updating skylib, the @addition_distfiles repository needs to be updated as well. Do so now. To be cherry-picked for #6493, i.e., the bazel 0.21.0 release. Change-Id: I6bf813324298a9b2ca3cb2cac15a871eef9e46f9 PiperOrigin-RevId: 224323158

laurentlb · 2018-12-17T20:53:09Z

Done.

scripts/release/release.sh create --force_rc=4 0.21.0 cb9b2af 12b9646 7fc967c 798b9a9 dbe05df

https://releases.bazel.build/0.21.0/rc4/index.html

laurentlb · 2018-12-18T17:13:52Z

@dslomov When can we release it?
I believe it can be released tomorrow, is it okay? (or shall I wait after new year?)

dslomov · 2018-12-18T17:36:20Z

I think tomorrow is fine. Could you include links to migration-0.21 and breaking-change-0.21 in release announcement?

laurentlb · 2018-12-19T14:40:43Z

Released: https://github.com/bazelbuild/bazel/releases/tag/0.21.0

laurentlb · 2018-12-19T16:22:22Z

cc @vbatts @petemounce

Blog post: https://blog.bazel.build/2018/12/19/bazel-0.21.html

vbatts · 2018-12-19T16:31:20Z

On 19/12/18 11:22 -0500, Laurent Le Brun wrote: cc ***@***.*** ***@***.*** Blog post: [3]https://blog.bazel.build/2018/12/19/bazel-0.21.html

#6965

jin · 2018-12-19T18:08:30Z

0.21.0 documentation is up: https://docs.bazel.build/versions/0.21.0/bazel-overview.html

vbatts · 2018-12-19T20:50:34Z

fedora/centos build: https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/838751/

sergiocampama · 2018-12-20T16:17:12Z

when will this hit homebrew?

laurentlb · 2018-12-20T21:50:17Z

It should be updated (bazelbuild/homebrew-tap#35)

petemounce · 2018-12-20T22:32:31Z

Published to chocolatey.

ittaiz · 2018-12-21T22:23:46Z

is there a clear guideline on what to do about protobuf? Like @davido said 3.6.1.3 isn't even released.

davido · 2018-12-23T07:25:37Z

Like @davido said 3.6.1.3 isn't even released.

@ittaiz I said this 16 days ago. Meantime protobuf v3.6.1.3 was released. However, it is not a complete protobuf release. Particularly, there is no protobuf-java on Maven Central that tagged with that version. Nevertheless, rules_closure and Gerrit were upgraded to protobuf v3.6.1.3 and it seems to work. So the hope is that protobuf 3.6.1.3 and protobuf-java 3.6.1 are binary compatible.

ittaiz · 2018-12-23T08:13:30Z

Thanks for the correction! Where was this released? If I look on their github project releases tab I don't see it.

…

On Sun, Dec 23, 2018 at 9:25 AM David Ostrovsky ***@***.***> wrote: Like @davido <https://github.com/davido> said 3.6.1.3 isn't even released. @ittaiz <https://github.com/ittaiz> I said this 16 days ago. Meantime protobuf v3.6.1.3 was released. However, it is not a complete protobuf release. Particularly, there is no protobuf-java on Maven Central that tagged with that version. Nevertheless, rules_closure and Gerrit were upgraded to protobuf v3.6.1.3 and it seems to work. So the hope is that protobuf 3.6.1.3 and protobuf-java 3.6.1 are binary compatible. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6493 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF00Mif9WNYDOZ8qsTcmu6yt1aElkks5u7y_7gaJpZM4X3r5w> .

davido · 2018-12-23T15:51:28Z

Where was this released?

Check this PR that updated rules_closure to protobuf 3.6.1.3: [1]:

http_archive(
        name = "com_google_protobuf",
        strip_prefix = "protobuf-3.6.1.3",
        sha256 = "73fdad358857e120fd0fa19e071a96e15c0f23bb25f85d3f7009abfd4f264a2a",
        urls = [
            "https://mirror.bazel.build/github.com/google/protobuf/archive/v3.6.1.3.tar.gz",
            "https://github.com/protocolbuffers/protobuf/archive/v3.6.1.3.tar.gz",
        ],
    )

Important: protobuf-java must still point to 3.6.1, like it's still currently done in gerrit:

maven_jar(
    name = "protobuf",
    artifact = "com.google.protobuf:protobuf-java:3.6.1",
    sha1 = "0d06d46ecfd92ec6d0f3b423b4cd81cb38d8b924",
)

[1] Bump protobuf to 3.6.1.3 rules_closure#320

davido · 2019-01-09T23:49:52Z

Bazel 0.21 package for Alpine Linux is here: https://github.com/davido/bazel-alpine-package/releases/tag/0.21.0.

petemounce · 2019-01-21T21:20:43Z

Belatedly, pushed to chocolatey.

laurentlb added the release label Oct 24, 2018

laurentlb self-assigned this Oct 24, 2018

laurentlb changed the title ~~Release 0.20 - December 2018~~ Release 0.21 - December 2018 Oct 24, 2018

laurentlb closed this as completed Dec 19, 2018

Release 0.21 - December 2018 #6493

Release 0.21 - December 2018 #6493

Comments

laurentlb commented Oct 24, 2018

laurentlb commented Dec 3, 2018 • edited Loading

davido commented Dec 3, 2018 • edited Loading

jin commented Dec 3, 2018

davido commented Dec 4, 2018 • edited Loading

jin commented Dec 4, 2018

davido commented Dec 4, 2018 • edited Loading

laurentlb commented Dec 4, 2018

meteorcloudy commented Dec 4, 2018

jin commented Dec 4, 2018 via email

lberki commented Dec 5, 2018

lberki commented Dec 5, 2018

davido commented Dec 5, 2018 • edited Loading

davido commented Dec 6, 2018 • edited Loading

laurentlb commented Dec 6, 2018

meteorcloudy commented Dec 6, 2018

jin commented Dec 7, 2018 • edited Loading

davido commented Dec 7, 2018 • edited Loading

laszlocsomor commented Dec 7, 2018

dslomov commented Dec 7, 2018

davido commented Dec 7, 2018 • edited Loading

laurentlb commented Dec 14, 2018

aehlig commented Dec 15, 2018

laurentlb commented Dec 17, 2018

laurentlb commented Dec 18, 2018

dslomov commented Dec 18, 2018

laurentlb commented Dec 19, 2018

laurentlb commented Dec 19, 2018

vbatts commented Dec 19, 2018 via email

jin commented Dec 19, 2018

vbatts commented Dec 19, 2018

sergiocampama commented Dec 20, 2018

laurentlb commented Dec 20, 2018

petemounce commented Dec 20, 2018

ittaiz commented Dec 21, 2018

davido commented Dec 23, 2018

ittaiz commented Dec 23, 2018 via email

davido commented Dec 23, 2018 • edited Loading

davido commented Jan 9, 2019

petemounce commented Jan 21, 2019

laurentlb commented Dec 3, 2018 •

edited

Loading

davido commented Dec 3, 2018 •

edited

Loading

davido commented Dec 4, 2018 •

edited

Loading

davido commented Dec 4, 2018 •

edited

Loading

davido commented Dec 5, 2018 •

edited

Loading

davido commented Dec 6, 2018 •

edited

Loading

jin commented Dec 7, 2018 •

edited

Loading

davido commented Dec 7, 2018 •

edited

Loading

davido commented Dec 7, 2018 •

edited

Loading

davido commented Dec 23, 2018 •

edited

Loading