release: 20.1.3

[asubiotto]

Candidate SHA: 7fd454f
Deployment status: Qualifying
Qualification Suite: https://teamcity.cockroachdb.com/viewType.html?buildTypeId=Cockroach_ReleaseQualification&tab=buildTypeStatusDiv&branch_Cockroach=provisional_202006230817_v20.1.3
Nightly Suite: https://teamcity.cockroachdb.com/viewType.html?buildTypeId=Cockroach_Nightlies_NightlySuite&tab=buildTypeStatusDiv&branch_Cockroach_Nightlies=provisional_202006230817_v20.1.3

• Roachtest GCE failed due to quota issues. Running independently: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032995&buildTypeId=Cockroach_Nightlies_WorkloadNightly

Admin UI for Qualification Clusters:

• Old Version: http://asubiotto-release-v2012-92670dd-0001.roachprod.crdb.io:26258/#/metrics/overview/cluster
• New Version: http://asubiotto-release-v2013-7fd454f-0001.roachprod.crdb.io:26258/#/metrics/overview/cluster

Release process checklist

Prep date: Tuesday 6/23/2020

• Pick a SHA
  • fill in Candidate SHA above
  • email thread on releases@
• Tag the provisional SHA
• Publish provisional binaries
• Ack security@ on the generated Stackdriver Alert to confirm these writes were part of a planned release (Just reply on the email received alert email acking that this was part of the release process)

Release Qualification

• Check binaries
  • Update VERSION to 20.1.3 with 7fd454f bincheck#203
• Deploy to test clusters
• Verify node crash reports
  • https://sentry.io/organizations/cockroach-labs/issues/1605608280/?project=164528&query=panic&statsPeriod=14d
• Start release qualification suite
• Start nightly suite
• fill in all WIP elements above (Nightly Suite, Admin UI, etc.)

One day after prep date:

• Get signoff on roachtest failures

• Keep an eye on clusters until release date. Do not proceed below until the release date.

Release date: Monday 6/29/2020

• Check cluster status

• Tag release

• Bless provisional binaries

• If applicable, update the map that roachtests use to map a version to a previous version, to reference the newly tagged version

  • roachtest: update version map and create fixtures #50770

• For production or stable releases in the latest major release series

  • Update the Homebrew Formula
    • bump version to 20.1.3 homebrew-tap#45
  • Update the image tag in our Orchestrator configurations
    • cloud: update orchestrator configs to point to v20.1.3 image #50752

• For production or stable releases

  • Announce version to the registration cluster

• Update docs

  • https://www.cockroachlabs.com/docs/releases/v20.1.3.html

• External communications for release

• Clean up provisional tag from repository

[asubiotto]

Starting the test failure checkoff process while the build is still running to save time.

Test Failures List

Roachtest GCE

Failures: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032995&buildTypeId=Cockroach_Nightlies_WorkloadNightly

[storage]

• clearrange/checks=true

[kv]

• gossip/chaos/nodes=9
• jepsen/sets/parts-start-kill-2
• transfer-leases/drain
• transfer-leases/quit

[appdev]

• django
• lib/pq
• pgx

[bulkio]

• import/tpch/nodes=8
• restore2TB/nodes=10

Random Syntax Tests

Failures: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032054&tab=buildResultsDiv&buildTypeId=Cockroach_Nightlies_RandomSyntaxTests

[sql-schema]

• TestRandomSyntaxSchemaChangeColumn

SQLite Logic Test High VModule Nightly

Failures: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032052&tab=buildResultsDiv&buildTypeId=Cockroach_Nightlies_SqlLogicTestHighVModuleNightly

[sql]

• Test parsing error tracked in logictest: SQLite Logic Test High VModule Nightly fails due to test output parsing error #50136

[tbg]

in mixed-versions, We weren't able to execute ./cockroach-v19.2.7, not sure why. Usually chmod +x ${FILE} is found to not have been run in this case.

   2: done
   3: done
   4: done
10:51:43 test.go:190: test status: 
10:51:43 test.go:190: test status: starting cluster
10:51:43 cluster.go:348: > /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --binary=./cockroach-19.2.7 --encrypt=false --sequential=false teamcity-2032053-1592909135-13-n4cpu4:1-4
teamcity-2032053-1592909135-13-n4cpu4: starting
0: ~ ././cockroach-19.2.7 version: exit status 126

[asubiotto]

This roachtest build also ran into CPU quota issues so I'll unfortunately have to restart it. The quota has been increased from 4k to 6k in us-central1 (although I'll still run this new run in us-east1) so this'll hopefully not happen again: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032995&buildTypeId=Cockroach_Nightlies_WorkloadNightly

[tbg]

gossip/chaos: infra flake

dial tcp 34.70.82.23:26257: connect: connection timed out

jepsen: infra flake

test_runner.go:725: failed to extend cluster: roachprod extend failed: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod extend teamcity-2032995-1592933600-42-n6cpu4 --lifetime=10m: exit status 1

The quit tests failed because they exceeded the one minute timeout in restartNode():

19:31:33 cluster.go:348: > /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --env=COCKROACH_SCAN_MAX_IDLE_TIME=5ms -a --vmodule=store=1,replica=1,replica_proposal=1 teamcity-2032995-1592933600-22-n3cpu4:2
teamcity-2032995-1592933600-22-n3cpu4: starting...........................................................19:32:33 test.go:325: test failure: 	cluster.go:1969,quit.go:96,quit.go:88,quit.go:142,context.go:135,quit.go:141,quit.go:88,quit.go:46,quit.go:340,test_runner.go:753: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start --env=COCKROACH_SCAN_MAX_IDLE_TIME=5ms -a --vmodule=store=1,replica=1,replica_proposal=1 teamcity-2032995-1592933600-22-n3cpu4:2 returned: context deadline exceeded

n2's logs show no signs of the node actually trying to start (the log is still that from previously gracefully shutting down the node), so I would call this an infra flake and am signing off.

[petermattis]

clearrange/checks=true infra strangeness:

19:31:31 cluster.go:348: > /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod stop teamcity-2032995-1592933600-30-n10cpu4
teamcity-2032995-1592933600-30-n10cpu4: stopping and waiting.................................................................................................................................
6: exit status 255: 
I200623 19:33:41.767276 1 cluster_synced.go:1749  command failed

I'm not quite sure what is going on yet, but this wasn't a failure of the test itself.

[rafiss]

Signed off on appdev tests. We have just pinned the versions of ORMs under test so after we update the list of expected failures, they will be more stable.

[pbardea]

import/tpch/nodes=8: infra flake

monitor task failed: dial tcp 34.71.144.14:26257: connect: connection timed out

restore2TB/nodes=10

monitor task failed: dial tcp 35.224.219.136:26257: connect: connection timed out