release: 20.2.0-alpha.2

[celiala]

Candidate SHA: d3119926d33d808c6384cf3e99a7f7435f395489 (d311992)
Tag: provisional_202007220233_v20.2.0-alpha.2

Deployment status: Qualifying
Qualification Suite: (passed ✅) Tests passed: 1, ignored: 291
Nightly Suite: (completed ✅) Tests failed: 24 (5 new), passed: 249, ignored: 19

Admin UI for Qualification Clusters:

Time	20.2.0-alpha.1	20.2.0-alpha.2
7/21, 11:45p	alpha.1	alpha.2

Release process checklist

Prep date: Mon 7/27 Thur 7/16, Thur 7/9, Mon 7/6 (usually a week before the release)

• Pick a SHA
  • fill in Candidate SHA above
  • email thread on releases@
• Tag the provisional SHA
• Publish provisional binaries
• Ack security@ on the generated Stackdriver Alert to confirm these writes were part of a planned release (Just reply on the email received alert email acking that this was part of the release process)

Release Qualification

• [Check binaries]
• Deploy to test clusters
• Verify node crash reports
• Start release qualification suite
• Start nightly suite
• fill in all WIP elements above (Nightly Suite, Admin UI, etc.)

One day after prep date:

• Get signoff on roachtest failures

• Keep an eye on clusters until release date. Do not proceed below until the release date.

Release date: WIP fill in date: usually Monday WIP

• Check cluster status

• Tag release

• Bless provisional binaries

• If applicable, update the map that roachtests use to map a version to a previous version, to reference the newly tagged version

• For production or stable releases in the latest major release series

  • Update the Homebrew Formula
  • Update the image tag in our Orchestrator configurations

• For production or stable releases

  • Announce version to the registration cluster

• Update docs

• External communications for release

Cleanup:

• Clean up provisional tag from repository
• Destroy roachprod clusters

[blathers-crl]

Hi @celiala, please add a C-ategory label to your issue. Check out the label system docs.

While you're here, please consider adding an A- label to help keep our repository tidy.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[celiala]

A release blocker for the alpha has come up in #50687.

We're going to need to cherry pick the fix: #51143

[celiala]

🚫 Deprecated (from older candidate SHA) 🚫

🚫 Roachtest Nightly - GCE

(Tests failed: 27 (8 new), passed: 245, ignored: 18)

🚫 Failures:
https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_WorkloadNightly/2080704

🚫 [appdev]

• django | [appdev]
• pgjdbc | [appdev]
• psycopg | [appdev]

🚫 [bulkio]

• backup2TB/n10cpu4 | [bulkio]
• dump-backwards-compatibility | [bulkio]
• import/tpcc/warehouses=1000/nodes=32 | [bulkio]
• tpccbench/nodes=6/cpu=16/multi-az | [kv bulkio]
• tpccbench/nodes=9/cpu=4/chaos/partition | [kv bulkio]
• tpccbench/nodes=9/cpu=4/multi-region | [kv bulkio]
• tpcc/mixed-headroom/n5cpu16 | [kv bulkio]
• drop/tpcc/w=100,nodes=9 | [kv bulkio]

🚫 [kv]

• kv50/rangelookups/relocate/nodes=8 | [kv]
• scaledata/jobcoordinator/nodes=6 | [kv]
• tpcc/interleaved/nodes=3/cpu=16/w=500 | [kv]
• follower-reads/nodes=3 | [kv]
• kv/contention/nodes=4 | [kv]
• kv/splits/nodes=3/quiesce=true | [kv]
• scaledata/filesystem_simulator/nodes=3 | [kv]
• scaledata/filesystem_simulator/nodes=6 | [kv]

🚫 [sql-exec]

• scrub/index-only/tpcc/w=100 | [sql-exec]
• sqlsmith/setup=empty/setting=no-mutations | [sql-exec]
• sqlsmith/setup=seed/setting=no-ddl | [sql-exec]
• scrub/all-checks/tpcc/w=100 | [sql-exec]
• sqlsmith/setup=seed/setting=no-mutations | [sql-exec]

🚫 [sql-schema]

• schemachange/index/tpcc/w=100 | [sql-schema]
• schemachange/during/tpcc | [sql-schema]
• schemachange/index/tpcc/w=1000 | [sql-schema]

🚫 Random Syntax Tests

Failures:
https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RandomSyntaxTests/2076005

🚫 [sql-exec]

• TestRandomSyntaxGeneration
• TestRandomSyntaxFunctions

🚫 SQLite Logic Test High VModule Nightly

🚫 Failures:
https://teamcity.cockroachdb.com/viewLog.html?buildId=2076003&tab=buildResultsDiv&buildTypeId=Cockroach_Nightlies_SqlLogicTestHighVModuleNightly

• 🚫 Exit code 1 (Step: make testlogic TESTFLAGS='--vmodule=*=10 -show-sql -test.v' (Command Line))

[ajwerner]

All of the schemachange test failures seem to be due to a timeout in the data import.
schemachange/index/tpcc/w=100 (1) - test_runner.go:804: test timed out (45m0s) - while running roachprod run teamcity-2080704-1594412253-72-n5cpu4:5 -- ./workload fixtures load tpcc --warehouses=100 {pgurl:1} and the same goes for schemachange/during/tpcc and schemachange/index/tpcc/w=1000

[irfansharif]

Checked off the KV failures. We're actively working on

• roachtest: kv/splits/nodes=3/quiesce=true failed #50865 (splits bug, being investigated by @aayushshah15)
• roachtest: kv/contention/nodes=4 failed #51028 (txnwait queue getting clogged up, being investigated by @nvanbenschoten), and
• roachtest: follower-reads/nodes=3 failed #50262 (follower reads bug, being investigated by @andreimatei)

There are a bunch of other "KV roachtests" that are failing due to wedged imports. I've reallocated them to bulk IO for investigation (I've assigned a bunch to @dt).

[rafiss]

Signed off on AppDev. Django and psycopg failures were because of parsing logic in the test that has since been fixed. PGJDBC has some flaky tests that we are figuring out how to ignore or stabilize.

[thoszhang]

Checking off the two schemachange/index/tpcc failures for reasons stated above. I'm taking another look at schemachange/during/tpcc.

[rohany]

Signing off on the RSG tests -- opened two issues to track the failures (#51472, #51473) but they shouldn't block the release.

[rohany]

Signing off on the SQLSmith failures. Opened some issues to track the individual failures but they shouldn't block the release.

[rohany]

The scrub failures have timed out due to the same reasons as the schemachange tests.

[celiala]

Roachtest Nightly - GCE

(Tests failed: 24 (5 new), passed: 249, ignored: 19)
https://teamcity.cockroachdb.com/viewLog.html?buildId=2107811&buildTypeId=Cockroach_Nightlies_WorkloadNightly

[appdev]

• django | [appdev]
• hibernate | [appdev]
• pgjdbc | [appdev]

[bulkio]

• dump-backwards-compatibility | [bulkio]
• jobs/mixed-versions | [bulkio]

[kv]

• acceptance/cli/node-status | [kv]
• acceptance/multitenant | [kv]
• autoupgrade | [kv]
• decommission-recommission | [kv]
• decommission/mixed-versions | [kv]
• decommission/nodes=4/duration=1h0m0s | [kv]
• follower-reads/nodes=3 | [kv]
• gossip/chaos/nodes=9 | [kv]
• scaledata/filesystem-simulator/nodes=6 | [kv]
• scaledata/job-coordinator/nodes=3 | [kv]
• scaledata/job-coordinator/nodes=6 | [kv]
• transfer-leases/drain | [kv]
• transfer-leases/quit | [kv]
• transfer-leases/signal | [kv]
• clearrange/checks=false | [kvstorage]
• clearrange/checks=true | [kvstorage]

[partitioning]

• roachmart/partition=true | [partitioning]

[storage]

Random Syntax Tests

https://teamcity.cockroachdb.com/viewLog.html?buildId=2107812&buildTypeId=Cockroach_Nightlies_RandomSyntaxTests

[sql-exec]

• TestRandomSyntaxGeneration
• TestRandomSyntaxFunctions

Other tests

Test	Message
SQLite Logic Tests	Process exited with code 2 (Step: Command Line)
Compose Tests	Process exited with code 1 (Step: Command Line)
SQLite Logic Test High VModule Nightly	Process exited with code 1 (Step: make testlogic TESTFLAGS='--vmodule=*=10 -show-sql -test.v' (Command Line))

[knz]

The " gossip/chaos/nodes=9 | [kv]" failure is #51721.

I believe it denotes a real bug but it's not new. I would tend to think it shouldn't block the release. @nvanbenschoten @andreimatei what do you think?

[knz]

The Random syntax generation tests are finding two real bugs #51829 + #51830. However I don't think they are release blockers. Signing off on them.

[rafiss]

Signed off on appdev tests. They aren't new problems.

[nvanbenschoten]

acceptance/multitenant is missing the --tenant-addr flag, which is due to the mismatch between the roachtest SHA and the CRDB SHA. That's not a blocker so signing off.

follower-reads/nodes=3 looks like a real issue. We see fewer than 2 follower reads occurred: saw [0 2 1] before and [0 3 1] after. @andreimatei and @aayushshah15 have been working in this area recently. They'll likely be interested in this issue.

[nvanbenschoten]

The " gossip/chaos/nodes=9 | [kv]" failure is #51721.

I believe it denotes a real bug but it's not new. I would tend to think it shouldn't block the release. @nvanbenschoten @andreimatei what do you think?

We see that nodes are getting different IDs from their roachprod IDs (see #51721 (comment)). It's unclear whether this is a regression inside cockroach or outside cockroach. We'll need to figure that out before we make a determination on whether this is a blocker or not.

[irfansharif]

It's not, that's me again. #51497 (comment), fixed by #51790.

[irfansharif]

Signing off the clearrange checks, looks to be infra-flakes: #51716, #51711.

[irfansharif]

Signed off the rest of the KV tests. The follower reads roachtest has been flaky for some time now, and still under investigation (so we shouldn't hold the release up). The autoupgrade test is another victim of roachprod --sequential being broken: #51776 (comment).

[solongordon]

Signed off on roachmart/partition=true. Looks like a timing issue on the test side and the test has passed in subsequent builds on master.