release: v19.2.0-rc.3

[solongordon]

Candidate SHA: 62801ce
Deployment status:

• rc2: http://solon-release-v1920-rc2-918d925b-0001.roachprod.crdb.io:26258
• rc3: http://solon-release-v1920-rc3-62801ce7-0001.roachprod.crdb.io:26258

Nightly Suite: https://teamcity.cockroachdb.com/viewLog.html?buildId=1571212

Release process checklist

Prep date: 2019-10-31

• Pick a SHA, fill in Candidate SHA above, notify #release-process of SHA.

• Tag the provisional SHA

• Publish provisional binaries
  https://teamcity.cockroachdb.com/viewLog.html?buildId=1568206

• Check binaries
  Update to v19.2.0-rc.3 bincheck#151

• Deploy to test clusters

• Verify node crash reports

• Start release qualification suite
  https://teamcity.cockroachdb.com/viewLog.html?buildId=1568237

• Start nightly suite

• fill in Deployment status above with clusters and Nightly Suite with the link to Nightly TeamCity Job

• Keep an eye on clusters until release date. Do not proceed below until the release date.

Release date: Monday 2019-11-04

• Check cluster status

• Tag release

• Bless provisional binaries

• For production or stable releases in the latest major release series

  • Update the Homebrew Formula
  • Update the image tag in our Orchestrator configurations
  • Announce version to the registration cluster
  • Upgrade the registration cluster (reg runs latest stable version)

• Update docs

• External communications for release

[solongordon]

Nightly roachtest failures (now OBSOLETE due to new SHA):

• cancel/tpcc/distsql/w=10,nodes=3 roachtest: cancel/tpcc/distsql/w=10,nodes=3 failed #38417, also failed in rc2
• clearrange/checks=true roachtest: clearrange/checks=true failed #38720, @ajwerner is investigating
• hibernate roachtest: hibernate failed #41751, cc @rafiss for sign-off
• import/tpch/nodes=8 roachtest: import/tpch/nodes=8 failed #39113
• kv50/rangelookups/relocate/nodes=8 roachtest: kv50/rangelookups/relocate/nodes=8 failed #41491, also failed in rc2
• scaledata/filesystem_simulator/nodes=3 roachtest: scaledata/filesystem_simulator/nodes=3 failed #36981, Looks consistent with historical flakiness
• tpccbench/nodes=9/cpu=4/multi-region roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #41876, @ajwerner signed off as an infra flake

[tbg]

Can't sign off on #38720, the error logs set off my alarms. #38720 (comment)

Asking someone else from the team to dig in.

[thoszhang]

I'm signing off on #39113 because the import failure due to a test timeout looks similar to the import slowness we're already aware of and that @dt has been investigating. More details here: #39113 (comment)

[solongordon]

Release process has been restarted with a new SHA to pick up #42068, which addresses the clearrange/checks=true roachtest failure above.

[solongordon]

Nightly roachtest failures:

• cancel/tpcc/distsql/w=10,nodes=3 roachtest: cancel/tpcc/distsql/w=10,nodes=3 failed #38417, also failed in rc2
• hibernate roachtest: hibernate failed #41751, cc @rafiss for sign-off. Determined to be mostly test flakes, with one actual failure (testCteNativeQueryOrdinalParameter) which was fixed on master but expected to still fail on 19.2.
• kv50/rangelookups/relocate/nodes=8 roachtest: kv50/rangelookups/relocate/nodes=8 failed #41491, also failed in rc2
• overload/tpcc_olap/nodes=3/cpu=8/w=50/c=96 roachtest: overload/tpcc_olap/nodes=3/cpu=8/w=50/c=96 failed #40774, cc @ajwerner for sign-off
• schemachange/mixed/tpcc roachtest: schemachange/mixed/tpcc failed #40935, cc @lucy-zhang. This issue is called out in 19.2 release blockers list #40447 as "probably not a blocker" so I'm assuming this is acceptable.
• tpccbench/nodes=9/cpu=4/multi-region roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #41876, likely an infra flake like in the previous run