release: v19.1.5

[ajwerner]

Candidate SHA: 6b14c0a
Deployment status: Deployed to ajwerner-release-1914 and ajwerner-release-1915 at 2:32EDT on 9/23/2019
Nightly Suite: https://teamcity.cockroachdb.com/viewLog.html?buildId=1502389&buildTypeId=Cockroach_Nightlies_NightlySuite

Release process checklist

Prep date: 9/23/2019

• Pick a SHA, fill in Candidate SHA above, notify #release-process of SHA.

  • 6b14c0a

• Tag the provisional SHA

  • provisional_201909231358_v19.1.5

• Publish provisional binaries

  • https://teamcity.cockroachdb.com/viewLog.html?buildId=1501818&buildTypeId=Cockroach_PublishProvisionalArtifacts

• Check binaries

  • Update VERSION for 19.1.5 bincheck#140

• Deploy to test clusters

  • http://ajwerner-release-v1914-0001.roachprod.crdb.io:26258/#/overview/list
  • http://ajwerner-release-v1915-0001.roachprod.crdb.io:26258/#/overview/list

• Verify node crash reports

  • https://sentry.io/organizations/cockroach-labs/issues/1030778567

• Start release qualification suite

  • https://teamcity.cockroachdb.com/viewLog.html?buildId=1502286&buildTypeId=Cockroach_ReleaseQualification

• Start nightly suite

  • https://teamcity.cockroachdb.com/viewLog.html?buildId=1502389&buildTypeId=Cockroach_Nightlies_NightlySuite
  • Failures:
    • jepsen/multi-register/start-stop-2
      • Infra flake: roachprod extend failed: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod extend teamcity-1569266905-39-n6cpu4 --lifetime=6m: exit status 1
    • scaledata/filesystem_simulator/nodes=3
      • Not new: roachtest: scaledata/filesystem_simulator/nodes=3 failed #36981
    • tpccbench/nodes=6/cpu=16/multi-az
      • Infra flake: Error: restoring fixture: pq: importing 3047 ranges: importing span /Table/60/1/435{7/10/2649-9/10/1207}: fetching "453551745420623876.sst": Get https://storage.googleapis.com/cockroach-fixtures/workload/tpcc/version=2.1.0,fks=true,interleaved=false,seed=1,warehouses=5000/customer/453551745420623876.sst: status code 500 trying to fetch http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
    • hibernate
      • roachtest: update hibernate blacklist for SFU #41101, much offline discussion about failures to stay in sync with hibernate upstream. The failure is extra tests passing, not failing.
    • schemachange/mixed/tpcc
      • roachtest: schemachange/mixed/tpcc failed #40935 (comment)
    • tpccbench/nodes=9/cpu=4/chaos/partition
      • Timed out stuck in a chaos loop trying to prepare statements. Seems unrelated to changes in this release.

• fill in Deployment status above with clusters and Nightly Suite with the link to Nightly TeamCity Job

• Keep an eye on clusters until release date. Do not proceed below until the release date.

Release date: 9/30/2019

• Check cluster status

• Tag release

• Bless provisional binaries

  • https://teamcity.cockroachdb.com/viewLog.html?buildId=1514417

• For production or stable releases in the latest major release series

  • Update the Homebrew Formula
    * cockroach 19.1.5 Homebrew/homebrew-core#44732
    * Upgrade homebrew formula to v19.1.5 homebrew-tap#29
  • Update the image tag in our Orchestrator configurations
    * cloud/kubernetes: Update cockroach version to v19.1.5 #41221
  • Announce version to the registration cluster
  • Upgrade the registration cluster (reg runs latest stable version)

• Update docs

  • Add 19.1.5 to releases.yaml docs#5500

• External communications for release

[bdarnell]

For the homebrew formula of this version, we want to add a message to the caveats section about the upcoming move to our own tap:

Due to a license change, the cockroach package in homebrew-core will no longer be updated when CockroachDB 19.2 is released. Please switch to https://github.com/cockroachdb/homebrew-tap instead.

We also need to make a version bump in the homebrew-tap repo in addition to homebrew-core. Since you're not a mac user @ajwerner I can handle the brew PRs on monday.

[knz]

@rmloveland and I found out the SHA may have been picked in error. It's just 2 commits beyond provisional_201903251931_v19.1-rc.1, a commit from March 2019.

[ajwerner]

🤦‍♂️ I'm restarting all of this now. Sorry all!

Restarting at d6a7e59.

[ajwerner]

Rebooting this now at 6b14c0a targetting a 9/30 release date.

[ajwerner]

@bdarnell I took a stab at typing up the brew PR. Here's the commit:

ajwerner/homebrew-core@ce081d9

If that looks right I'll PR it and follow the same patterns on our tap.

[bdarnell]

👍 on submitting that PR upstream.

On our tap it'll be simpler (no caveat change) but a different SHA because it downloads the binary tarball instead of the source one.

[ajwerner]

This is still open due to the not-yet-merged Homebrew PR. I've gotten my hands on a mac and am working on it now.

[ajwerner]

The homebrew build seems to be broken due to rocksdb compilation problems:

In file included from /tmp/cockroach-20191005-36246-1n1kmgg/cockroach-v19.1.5/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/builder.cc:19:
/tmp/cockroach-20191005-36246-1n1kmgg/cockroach-v19.1.5/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/db/internal_stats.h:111:5: error: declaration shadows a variable in namespace 'rocksdb' [-Werror,-Wshadow]
    WAL_FILE_BYTES,
    ^
...

https://jenkins.brew.sh/job/Homebrew%20Core%20Pull%20Requests/49277/version=catalina/testReport/junit/brew-test-bot/catalina/install___build_bottle_cockroach/

@ajkr could I have some help with this later today?

[bdarnell]

The upstream fix is facebook/rocksdb#5779.

There's a workaround in https://github.com/cockroachdb/rocksdb/pull/41/files that would be easier to patch in via the homebrew formula.

[ajwerner]

Thanks @bdarnell!

The build was also broken due to the use of modules:

bin/prereqs: go/build: importGo golang.org/x/net/html: exit status 1
error writing go.mod: open /private/tmp/cockroach-20191007-76155-1qpr9si/cockroach-v19.1.5/pkg/mod/github.com/andybalholm/[email protected]/go.mod298498081.tmp: permission denied

I've updated https://github.com/Homebrew/homebrew-core/pull/44732/files to patch rocksdb/CMakeLists.txt and to set GO111MODULE=off.

I've verified that it now builds on a mac.

[ajwerner]

Ugh the build failed to build the bottle on Mojave and Catalina though not on High Sierra.

https://jenkins.brew.sh/job/Homebrew%20Core%20Pull%20Requests/49436/

/tmp/cockroach-20191009-18802-1c75inf/cockroach-v19.1.5/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/utilities/persistent_cache/block_cache_tier_file.h:265:9: error: explicitly defaulted copy assignment operator is implicitly deleted [-Werror,-Wdefaulted-function-deleted]
    IO& operator=(const IO&) = default;
        ^

I'm happy to add -Wno-defaulted-function-deleted to the patch which I suspect will fix the error.

@bdarnell or @ajkr thoughts on whether this is surprising or expected?

[bdarnell]

That is surprising; it was building on mojave before (and i don't see any reference to this in the rocksdb tracker). Maybe this is from the xcode update that was just released? It'll be nice to get back to our own tap and isolated build.

You could set the warning flag, or apply the upstream fix from facebook/rocksdb#5095

[ajwerner]

There was indeed a major version upgrade to AppleClang. I'm just going to suppress the error.

[ajwerner]

Due to my own confusion about how to safely upgrade the reg cluster from 19.1.4 to 19.1.5 I'm going to close this. If somebody tells me I need to do it I will go and figure it out. In the meantime I've asked for some guidance elsewhere.