Script for the PublishRelease TC build configuration #50053

jlinder · 2020-06-10T16:35:25Z

Before: the script wasn't implemented.

Now:

Part of the new release process, this script

tags the selected SHA to the provided name
compiles the binaries and archive and uploads them to S3 as the
versioned name
uploads the docker image to docker.io/cockroachdb/cockroach
pushes the tag to github.com/cockroachdb/cockroach
push all the artificats to their respective latest locations as
appropriate

Release note: None

cockroach-teamcity · 2020-06-10T16:35:32Z

This change is

otan · 2020-06-10T17:02:36Z

build/release/teamcity-publish-release.sh

+# Work around headless d-bus problem by forcing docker to use
+# plain-text credentials for dockerhub.
+# https://github.com/docker/docker-credential-helpers/issues/105#issuecomment-420480401
+mkdir -p ~/.docker


this section of docker login may make sense to modularise

jlinder · 2020-08-14T13:56:18Z

bors ping

craig · 2020-08-14T13:56:19Z

pong

jlinder · 2020-08-20T15:15:19Z

build/release/teamcity-publish-release.sh

+# for the `metadata` part at the end because Docker tags don't support
+# `+` in the tag name.
+# https://github.com/cockroachdb/cockroach/blob/master/pkg/util/version/version.go#L75
+build_name="$(echo "${NAME}" | grep -E -o '^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-[-.0-9A-Za-z]+)?$')"


I tried using +dryrun instead of .dryrun or -dryrun and found docker image tags don't allow + in the tag name. I chose to not allow + in the build name so this problem isn't encountered. This means we can't have releases (or even internal builds that we want CC to consume) with +'s in them. I'm open to alternative ways of solving this if you've any ideas.

jlinder · 2020-08-20T15:16:33Z

build/release/teamcity-publish-release.sh

+
+# TODO: update publish-artifacts with option to leave one or more cockroach binaries in the local filesystem
+curl -f -s -S -o- "https://${s3_download_hostname}/cockroach-${build_name}.linux-amd64.tgz" | tar ixfz - --strip-components 1
+cp cockroach build/deploy


Only copying cockroach (and not the libgeo libs) so this can be backported to v20.1 and before. Will make a follow-on PR to add libgeo for master.

jlinder · 2020-08-20T15:18:02Z

build/release/teamcity-publish-release.sh

+    AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
+    AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
+    TC_BUILD_BRANCH="$build_name" \
+    publish-provisional-artifacts -bless -release -bucket "${bucket}"


This is not copying / linking the files for dryrun but it should for binaries.cockroachdb.com since it is used in bless provisional artifacts already. I'll fix this post this PR.

jlinder · 2020-08-20T15:19:15Z

build/release/teamcity-publish-release.sh

+# example: v20.1-latest
+if [[ -z "$PRE_RELEASE" ]]; then
+  #TODO: implement me!
+  echo "Pushing latest-RELEASE_BRANCH S3 binaries and archive is not implemented."


Since this is a new feature and not necessary for this initial test, leaving this for implementation in a subsequent PR.

jlinder · 2020-08-20T15:20:48Z

build/release/teamcity-publish-release.sh

+  s3_download_hostname="${bucket}"
+  git_repo_for_tag="cockroachdb/cockroach"
+else
+  bucket="${BUCKET:-cockroach-builds-test}"


What do you all think about using the binaries-test.cockroachdb.com here? (I'm not sure what the history is of the bucket.)

jlinder · 2020-08-20T15:37:44Z

PTAL.

Successful builds:

pbardea

LGTM - with minor comments

pbardea · 2020-08-20T16:16:48Z

build/release/teamcity-publish-release.sh

+# Matching the version name regex from within the cockroach code except
+# for the `metadata` part at the end because Docker tags don't support
+# `+` in the tag name.
+# https://github.com/cockroachdb/cockroach/blob/master/pkg/util/version/version.go#L75


I'm worried about this link rotting - maybe using the permalink (

cockroach/pkg/util/version/version.go

Line 75 in 4c6864b

`^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-[0-9A-Za-z-.]+)?(\+[0-9A-Za-z-.]+|)?$`,

) would at least allow future readers to trace back the history if this changes?

Indeed. Updated.

pbardea · 2020-08-20T16:17:43Z

build/release/teamcity-publish-release.sh

+#                                         ^major           ^minor           ^patch         ^preRelease
+
+if [[ -z "$build_name" ]] ; then
+    echo 'Invalid NAME. Must be of the format "vMAJOR.MINOR.PATCH(-PRERELEASE)?".'


Should we also print the NAME that we're trying to parse here?

That would be helpful. Added.

pbardea · 2020-08-20T16:18:22Z

build/release/teamcity-publish-release.sh

+    exit 1
+fi
+
+release_branch=$(echo ${build_name} | grep -E -o '^v[0-9]+\.[0-9]+')


Do we need to check if we've successfully parse the release_branch here?

I didn't think so because this is a subset of what is checked for build_name and if the build_name fails then it won't make it here.

jlinder · 2020-08-20T19:17:33Z

TFTR!

bors r=pbardea

craig · 2020-08-20T19:17:36Z

🕐 Waiting for PR status (Github check) to be set, probably by CI. Bors will automatically try to run when all required PR statuses are set.

Before: the script wasn't implemented. Now: Part of the new release process, this script - tags the selected SHA to the provided name - compiles the binaries and archive and uploads them to S3 as the versioned name - uploads the docker image to docker.io/cockroachdb/cockroach - pushes the tag to github.com/cockroachdb/cockroach - push all the artificats to their respective `latest` locations as appropriate Release note: None

craig · 2020-08-20T19:27:41Z

👎 Rejected by PR status

jlinder · 2020-08-20T19:29:32Z

bors r=pbardea

craig · 2020-08-20T21:12:12Z

Build succeeded:

GitHub CI (Cockroach)

jlinder requested review from otan and dt June 10, 2020 16:50

otan reviewed Jun 10, 2020

View reviewed changes

jlinder mentioned this pull request Jul 3, 2020

Release: Automate the release process #50955

Closed

7 tasks

jlinder force-pushed the release/teamcity-publish-release branch from 5d5463b to 4d29a21 Compare August 6, 2020 03:24

blathers-crl bot requested a review from otan August 6, 2020 03:24

jlinder requested a review from pbardea August 6, 2020 15:31

jlinder force-pushed the release/teamcity-publish-release branch from 4d29a21 to 663a76c Compare August 6, 2020 15:51

jlinder mentioned this pull request Aug 6, 2020

release: add script to add qualification tags to release #52454

Merged

jlinder force-pushed the release/teamcity-publish-release branch from 663a76c to ae88bb9 Compare August 13, 2020 22:04

jlinder force-pushed the release/teamcity-publish-release branch 2 times, most recently from f62d96e to 5a06605 Compare August 20, 2020 15:14

jlinder commented Aug 20, 2020

View reviewed changes

jlinder force-pushed the release/teamcity-publish-release branch from 5a06605 to c5bf9b2 Compare August 20, 2020 15:33

jlinder marked this pull request as ready for review August 20, 2020 15:37

pbardea approved these changes Aug 20, 2020

View reviewed changes

jlinder force-pushed the release/teamcity-publish-release branch 2 times, most recently from 290c240 to 9d126ac Compare August 20, 2020 18:53

jlinder force-pushed the release/teamcity-publish-release branch from 9d126ac to d73710d Compare August 20, 2020 19:21

craig bot merged commit 979127c into cockroachdb:master Aug 20, 2020

This was referenced Aug 21, 2020

release-20.1: Script for the PublishRelease TC build configuration #53173

Merged

release-19.2: Script for the PublishRelease TC build configuration #53174

Merged

release-19.1: Script for the PublishRelease TC build configuration #53175

Merged

jlinder deleted the release/teamcity-publish-release branch August 25, 2020 18:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Script for the PublishRelease TC build configuration #50053

Script for the PublishRelease TC build configuration #50053

jlinder commented Jun 10, 2020

cockroach-teamcity commented Jun 10, 2020

otan Jun 10, 2020 •

edited

Loading

jlinder commented Aug 14, 2020

craig bot commented Aug 14, 2020

jlinder Aug 20, 2020

jlinder Aug 20, 2020

jlinder Aug 20, 2020

jlinder Aug 20, 2020

jlinder Aug 20, 2020

jlinder commented Aug 20, 2020

pbardea left a comment

pbardea Aug 20, 2020

jlinder Aug 20, 2020

pbardea Aug 20, 2020

jlinder Aug 20, 2020

pbardea Aug 20, 2020

jlinder Aug 20, 2020

jlinder commented Aug 20, 2020

craig bot commented Aug 20, 2020

craig bot commented Aug 20, 2020

jlinder commented Aug 20, 2020

craig bot commented Aug 20, 2020

Script for the PublishRelease TC build configuration #50053

Script for the PublishRelease TC build configuration #50053

Conversation

jlinder commented Jun 10, 2020

cockroach-teamcity commented Jun 10, 2020

otan Jun 10, 2020 • edited Loading

Choose a reason for hiding this comment

jlinder commented Aug 14, 2020

craig bot commented Aug 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlinder commented Aug 20, 2020

pbardea left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlinder commented Aug 20, 2020

craig bot commented Aug 20, 2020

craig bot commented Aug 20, 2020

jlinder commented Aug 20, 2020

craig bot commented Aug 20, 2020

otan Jun 10, 2020 •

edited

Loading