release: 19.1.10

[asubiotto]

Candidate SHA: 7101af9
Deployment status: Qualifying
Qualification Suite: https://teamcity.cockroachdb.com/viewType.html?buildTypeId=Cockroach_ReleaseQualification&branch_Cockroach=provisional_202006230815_v19.1.10&tab=buildTypeStatusDiv
Nightly Suite: https://teamcity.cockroachdb.com/viewType.html?buildTypeId=Cockroach_Nightlies_NightlySuite&branch_Cockroach_Nightlies=provisional_202006230815_v19.1.10&tab=buildTypeStatusDiv

• Roachtest GCE failed due to quota issues. Running independently: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032556&tab=queuedBuildOverviewTab

Admin UI for Qualification Clusters:

• Old Version: http://alfonso-release-v1919-d556976-0001.roachprod.crdb.io:26258/#/metrics/overview/cluster
• New Version: http://alfonso-release-v19110-7101af9-0001.roachprod.crdb.io:26258/#/metrics/overview/cluster

Release process checklist

Prep date: Monday 6/22/2020

• Pick a SHA
  • fill in Candidate SHA above
  • email thread on releases@
• Tag the provisional SHA
• Publish provisional binaries
• Ack security@ on the generated Stackdriver Alert to confirm these writes were part of a planned release (Just reply on the email received alert email acking that this was part of the release process)

Release Qualification

• Check binaries
  • Update v19.1.10 VERSION to 7101af9 bincheck#201
• Deploy to test clusters
• Verify node crash reports
  • https://sentry.io/organizations/cockroach-labs/issues/?project=164528&query=panic&statsPeriod=14d
• Start release qualification suite
• Start nightly suite
• fill in all WIP elements above (Nightly Suite, Admin UI, etc.)

One day after prep date:

• Get signoff on roachtest failures

• Keep an eye on clusters until release date. Do not proceed below until the release date.

Release date: Monday 6/29/2020

• Check cluster status

• Tag release

• Bless provisional binaries

• If applicable, update the map that roachtests use to map a version to a previous version, to reference the newly tagged version

  • roachtest: update version map and create fixtures #50770

• For production or stable releases in the latest major release series

  • Update the Homebrew Formula
  • Update the image tag in our Orchestrator configurations

• For production or stable releases

  • Announce version to the registration cluster

• Update docs

  • https://www.cockroachlabs.com/docs/releases/v19.1.10.html

• External communications for release

• Clean up provisional tag from repository

[asubiotto]

Publish bleeding edge for the selected sha (73a373f at time of writing) seems to have failed: test_driver.go:232: error creating order: HTTP error 500: 500 Internal Server Error. The diff from the last successful run doesn't seem suspicious (fixing opt join simplification), so I suspect this might be a flake. Manually triggered another run: https://teamcity.cockroachdb.com/viewType.html?buildTypeId=Cockroach_MergeToMaster&tab=buildTypeChains&branch_Cockroach=release-19.1#_expand=block_bt133_bt151_bt154_bt17-2027662&hpos=0&vpos=113

[asubiotto]

This run failed as well. But I was confused before. Looks like we did have a successful run on the 31 May with this sha (https://teamcity.cockroachdb.com/viewLog.html?buildId=1980371&buildTypeId=Cockroach_MergeToMaster). @rafiss it's possible this failure is due to a test issue. It looks like there were a bunch of updates to ActiveRecord on June 9th (https://github.com/cockroachdb/examples-orms/commits/master) and the first failed test is on June 10th.

I'm going to go ahead with this SHA (73a373f), hoping the latest red build is not a problem.

[asubiotto]

@jlinder looks like there was also a problem posting a failure for this issue because of an unset GITHUB_API_TOKEN:

[05:32:32]Traceback (most recent call last):
[05:32:32]  File "build/teamcity-post-failures.py", line 175, in <module>
[05:32:32]    post_issue(issue)
[05:32:32]  File "build/teamcity-post-failures.py", line 133, in post_issue
[05:32:32]    headers={'Authorization': 'token {0}'.format(os.environ['GITHUB_API_TOKEN'])})
[05:32:32]  File "/usr/lib/python3.6/os.py", line 669, in __getitem__
[05:32:32]    raise KeyError(key) from None
[05:32:32]KeyError: 'GITHUB_API_TOKEN'
[05:32:32]Process exited with code 1

[asubiotto]

I think there are going to be problems with the nightlies until we resolve the Examples-ORMs failure since this is a dependency for the roachtests.

[rafiss]

I'm looking into examples-orms. It has been passing against master, so somehow the problem seems to be specific to the 19.1 branch.

[jlinder]

@jlinder looks like there was also a problem posting a failure for this issue because of an unset GITHUB_API_TOKEN:

It looks like the GITHUB_API_TOKEN was mistakenly removed back in December. I've added it back in as an env. parameter.

[rafiss]

Here is a PR to get examples-orms passing: cockroachdb/examples-orms#106. We'll skip the test for versions <=19.1 until we can debug the underlying issue. (Note, we may decide to leave the skipped test permanently, as our tooling support page specifies that we support ActiveRecord for CockroachDB 20.1+.)

[asubiotto]

Updated sha with security fix: 7101af9

[asubiotto]

@RaduBerinde looks like the random syntax tests failed in the nightlies. This isn't a blocker but it looks like a query is stuck in optimization. The symptoms look similar to #43076. Was the fix for that backported to 19.1? Here is the link if you want to take a look: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032012&buildTypeId=Cockroach_Nightlies_RandomSyntaxTests&tab=buildResultsDiv&branch_Cockroach_Nightlies=provisional_202006230815_v19.1.10

[asubiotto]

Urgh, looks like we ran into the same quota issue as #50465 (comment). Will have to rerun the Roachtest GCE build: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032556&tab=queuedBuildOverviewTab

[asubiotto]

Starting the test failure checkoff process while the build is still running to save time.

Test Failures List

Roachtest GCE

Failures: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032556&tab=queuedBuildOverviewTab

[storage]

• clearrange/checks=false

[appdev]

• hibernate
• psycopg

[kv]

• kv0bench/nodes=10/cpu=8/shards=20/sequential
• ycsb/B/nodes=3/cpu=32
• tpccbench/nodes=9/cpu=4/chaos/partition

[bulk-io]

• restore2TB/nodes=10

[sql-schema]

• schemachange/random-load

Random Syntax Tests

Failures: https://teamcity.cockroachdb.com/viewLog.html?buildId=2032012&buildTypeId=Cockroach_Nightlies_RandomSyntaxTests&tab=buildResultsDiv&branch_Cockroach_Nightlies=provisional_202006230815_v19.1.10

[optimizer]

• TestRandomSyntaxSQLSmith

[rafiss]

Signed off on hibernate -- we recently expanded the test coverage, but had missed marking some expected failures for 19.1

[maddyblue]

Signed off on sqlsmith. Statement timeout, which happens a bunch.

[petermattis]

Signed off on clearrange/checks=false. The failure is a dup of #44845. We run out of disk space one one node during the IMPORT phase of the test. This has been happening intermittently on the various release branches for a while now.

[tbg]

Signed off on the kv tests. One of them looked like a straight-up infra problem. The other two have trouble (re)starting CRDB and return with an opaque error from roachprod, in both cases the logs look like the node actually did start. I would dig deep into this on master or 20.1, but since this is 19.1 I'm fine chalking this up to some infra problem as well, though less sure.

[rafiss]

Signed off on psycopg2 -- it's hanging because of a bug in the tests that has been fixed upstream

[spaskob]

signed off for schema change random load

[pbardea]

restore2TB/nodes=10: infra flake during health check, but node appears healthy

monitor task failed: dial tcp 35.226.173.7:26257: connect: connection timed out