release: v20.2.0-beta.1

[nathanstilwell]

Release Details

Candidate SHA: 32525b00aeb71f1cd595320228a21206aada183e
Deployment status: PREPARING
Qualify Build: build
Nightly Suite: build
Admin UI for Qualification Clusters: Old Cluster Admin // New Cluster Admin
Working Tags
- Provisional Tag: provisional_202008252111_v20.2.0-beta.1
- Release Staging Tag: v20.2.0-alpha.1-2759-g32525b00ae

Restarting with new SHA

Candidate SHA: eaa939ce6548a54a23970814ff00f30ad87680ac
Deployment status: PREPARING
Qualify Build: build
Nightly Suite: build
Admin UI for Qualification Clusters: Old Cluster Admin // New Cluster Admin

Working Tags

• Provisional Tag: provisional_202008261913_v20.2.0-beta.1
• Release Staging Tag: v20.2.0-alpha.1-2795-geaa939ce65

Process List

Release process checklist

Prep date: 25 AUG

• Pick a SHA

  • fill in Candidate SHA above
  • email thread on releases@

• Tag the provisional SHA

• Run Make and Publish

Release Qualification

• Start nightly suite

• Deploy to test clusters
  Old cluster :: nathanstilwell-release-v2020-alpha3-a2c6646c
  New cluster :: nathanstilwell-release-v2020-beta1-32525b00

• Verify node crash reports

  • Resolve Crash Report Report
  • Close Panic Issue

For new release binaries use commands...

export VERSION=<Release Staging Tag>
curl https://cockroach-builds.s3.amazonaws.com/cockroach-${VERSION}.darwin-10.9-amd64.tgz | tar -xJ
echo "select crdb_internal.force_panic('testing');" | ./cockroach-${VERSION}.darwin-10.9-amd64/cockroach demo --empty

• fill in all WIP elements above (Nightly Suite, Admin UI, etc.)

One day after prep date:

• Get signoff on roachtest failures

• Keep an eye on clusters until release date. Do not proceed below until the release date.

Release date: 31 AUG

• Check cluster status (Regular Release Process)

• Publish Cockroach Release (Updated process, see notes)

Go to TeamCity and run Publish Cockroach Release using the Release Staging Tag above on the changes tab, and in parameters enter v20.2.0-beta.1 in the Version Name field, and check the box for Pre-release?.

• Ack security@ and release-engineering-team@ on the generated AWS S3 bucket write Alert to confirm these writes were part of a planned release (Just reply on the email received alert email acking that this was part of the release process)

The Publish Cockroach Release job will publish binaries to S3 triggering alert emails. The content will be a blob of unformatted json. Inside of the JSON there will be an object property for each binary written to S3. It looks something like this,

        "object": {
          "key": "cockroach-v20.2.0-alpha.3.linux-amd64.tgz",
          "size": 91943077,
          "eTag": "d4a3c6ea1747b9ee97a0e1a305117698",
          "sequencer": "005F4456A334ED2A28"
        }

There will be 4 emails sent for each of the binaries, so look out for emails for linux, windows, darwin, and src. Reply to the message chain stating that these writes were part of a release.

• Check binaries (Regular Release Process)

• Check Docker Img (Regular Release Process, but here's a hint)

Run this command from a shell and check that the version and SHA match above

> docker run cockroachdb/cockroach-unstable:v20.2.0-beta.1 version

• Update docs
• External communications for release

Cleanup:

• Clean up provisional tag from repository
• Destroy roachprod clusters

[nathanstilwell]

Roachtest Nightly - GCE: Test Failure Sign-Off

Roachtest Nightly - GCE: Build Link

Build 1989 at 26 Aug 20:29
Tests failed: 13 (4 new), passed: 253, ignored: 27

appdev

• activerecord
• django
• pgjdbc

bulkio

• restore2TB/nodes=32
• dump-backwards-compatibility

cdc

• cdc/poller/rangefeed=false
• cdc/tpcc-1000

kv

• kv/contention/nodes=4
• jepsen/multi-register/parts-start-kill-2
• decommission/mixed-versions
• decommission/randomized
• tpcc/mixed-headroom/n5cpu16
• tpccbench/nodes=9/cpu=4/multi-region

[rafiss]

Signed off on appdev tests. We have PRs open to skip those flaky cases.

[adityamaru]

Signing off on dump-backwards-compatibility. The roachtest is failing because we have deprecated the --insecure flag, I am working on updating it.
Signing off on restore2TB/nodes=32. This is an ongoing investigation with @pbardea's help, but this is not a regression since the last release.

[tbg]

Signed off on the decommission tests (the tests are new and need deflaking), and the jepsen test (timed out, but somewhere in the jepsen guts, and not after announcing an failure).

I asked @nvanbenschoten to look into tpccbench, which has a concerning crash in the concurrency manager, and @andreimatei for tpcc/mixed-headroom which has a concerning missing row in the stock table (the last time we saw this, it was a RocksDB bug - sure hope it's easier this time). Both of these could possibly block the release, so we need to double down on them now.

[RaduBerinde]

Closing this. New tracking issue #54059.