chore(ci): GHA - artifact build and push to GAR & GCS (#329)

* chore(ci): GHA - container image build and push

* fix(ci): for PR's build container without GAR login

* fix(ci): dont't run GHA build on forks

Each time a fork's master branch is resync'd with spinnaker/<repo> we
don't want to trigger GHA CI.

* chore(ci): GHA - Build and Publish spin binaries to GCS

Maintain existing behaviours:

shared:
- build for darwin, linux, windows, all on amd64, we can add aarch64 if
  wish.

on release:
- publish to `spinnaker-artifacts/spin/<release>/<os>/<arch>/spin(.exe)`
- publish `spinnaker-artifacts/spin/latest` text file with new version
  if version is newest (version sort). eg: 1.23.0 newer than 1.22.5.
- publish `spinnaker-artifacts/spin/<MAJOR.MINOR>.x-latest` text file
  with version number, eg: 1.23.0 or 1.23.1. Assuming we always do
  increment patch versions.

on merge master or release-* branch:
- publish to `spinnaker-artifacts/spin/<branch>/<git sha>-<datetime>/<os>/<arch>/spin(.exe)`
- this is different see next paragraph

Changes from previous behaviour:

Previous builds were always `spinnaker-artifacts/spin/<tag>-<datetime>/<os>/<arch>/spin(.exe)`
except for final release versions (1.22.0).
eg:
```
1.17.3-20201209030017/
1.17.3-20201210030017/
1.17.3/
```

However I believe it is easier to sort and review with the extra branch or
release number directory, eg:
```
master/<git-sha>-<datetime>/...
release-1.17.x/<git-sha>-<datetime>/...
```
This is also consistent with the container image tags now generated in GHA.

* chore(ci): Remove binary upload to GCS bucket script

This task has been moved into GHA per preceding commit.

* chore(ci): GHA - Test and build binaries before containers

We don't need extra job for test when `go test` is fast and we are
performing the same setup steps for `build` job.

Also Dockerfile doesn't have `go test` step.

* chore(ci): GHA - simplify build versioning

- collapse version info gathering steps into single `build_variables` step
- collapse version info parts into single string and use everywhere.
- use short git sha: `(git rev-parse --short HEAD)`

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

.github/workflows/build.yml

@@ -0,0 +1,92 @@
+name: Branch Build
+
+on:
+  push:
+    branches:
+    - master
+    - release-*
+
+env:
+  CONTAINER_REGISTRY: us-docker.pkg.dev/spinnaker-community/docker
+
+jobs:
+  lint:
+    # Only run this on repositories in the 'spinnaker' org, not on forks.
+    if: startsWith(github.repository, 'spinnaker/')
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Ensure code formatting and style is consistent
+      uses: golangci/golangci-lint-action@v2
+      with:
+        version: v1.32
+
+  branch-build:
+    # Only run this on repositories in the 'spinnaker' org, not on forks.
+    if: startsWith(github.repository, 'spinnaker/')
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Prepare build variables
+        id: build_variables
+        run: |
+          echo ::set-output name=REPO::${GITHUB_REPOSITORY##*/}
+          echo ::set-output name=VERSION::"${GITHUB_REF_NAME}-$(git rev-parse --short HEAD)-$(date --utc +'%Y%m%d%H%M')"
+          echo ::set-output name=VERSION_WITHOUT_BRANCH::"$(git rev-parse --short HEAD)-$(date --utc +'%Y%m%d%H%M')"
+
+      - uses: actions/setup-go@v2
+        with:
+          go-version: 1.15
+      - uses: actions/cache@v2
+        with:
+          path: ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-
+      - name: Fetch dependencies
+        run: go get -d -v
+      - name: Test
+        run: go test -v ./...
+      - name: Build binaries
+        env:
+          LDFLAGS: "-X github.com/spinnaker/spin/version.Version=${{ steps.build_variables.outputs.VERSION }}"
+        run: |
+          GOARCH=amd64 GOOS=darwin go build -ldflags "${LDFLAGS}" -o dist/${{ github.ref_name }}/${{ steps.build_variables.outputs.VERSION_WITHOUT_BRANCH }}/darwin/amd64/spin .
+          GOARCH=amd64 GOOS=linux go build -ldflags "${LDFLAGS}" -o dist/${{ github.ref_name }}/${{ steps.build_variables.outputs.VERSION_WITHOUT_BRANCH }}/linux/amd64/spin .
+          GOARCH=amd64 GOOS=windows go build -ldflags "${LDFLAGS}" -o dist/${{ github.ref_name }}/${{ steps.build_variables.outputs.VERSION_WITHOUT_BRANCH }}/windows/amd64/spin.exe .
+          dist/${{ github.ref_name }}/${{ steps.build_variables.outputs.VERSION_WITHOUT_BRANCH }}/linux/amd64/spin --version
+      - name: Login to Google Cloud
+        uses: 'google-github-actions/auth@v0'
+        # use service account flow defined at: https://github.com/google-github-actions/upload-cloud-storage#authenticating-via-service-account-key-json
+        with:
+          credentials_json: '${{ secrets.GAR_JSON_KEY }}'
+      - name: Upload spin CLI binaries to GCS
+        uses: 'google-github-actions/upload-cloud-storage@v0'
+        with:
+          path: 'dist/'
+          destination: 'spinnaker-artifacts/spin'
+          parent: false
+
+      - name: Login to GAR
+        # Only run this on repositories in the 'spinnaker' org, not on forks.
+        if: startsWith(github.repository, 'spinnaker/')
+        uses: docker/login-action@v1
+        # use service account flow defined at: https://github.com/docker/login-action#service-account-based-authentication-1
+        with:
+          registry: us-docker.pkg.dev
+          username: _json_key
+          password: ${{ secrets.GAR_JSON_KEY }}
+      - name: Build and publish container image
+        # Only run this on repositories in the 'spinnaker' org, not on forks.
+        if: startsWith(github.repository, 'spinnaker/')
+        uses: docker/build-push-action@v2
+        with:
+          context: .
+          file: Dockerfile
+          push: true
+          build-args: |
+            "VERSION=${{ steps.build_variables.outputs.VERSION }}"
+          tags: |
+            "${{ env.CONTAINER_REGISTRY }}/${{ steps.build_variables.outputs.REPO }}:${{ github.ref_name }}-latest"
+            "${{ env.CONTAINER_REGISTRY }}/${{ steps.build_variables.outputs.REPO }}:${{ steps.build_variables.outputs.VERSION }}"

.github/workflows/pr.yml

@@ -0,0 +1,61 @@
+name: PR Build
+
+on: [ pull_request ]
+
+env:
+  CONTAINER_REGISTRY: us-docker.pkg.dev/spinnaker-community/docker
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Ensure code formatting and style is consistent
+      uses: golangci/golangci-lint-action@v2
+      with:
+        version: v1.32
+
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Prepare build variables
+        id: build_variables
+        run: |
+          echo ::set-output name=REPO::${GITHUB_REPOSITORY##*/}
+          echo ::set-output name=VERSION::"$(git rev-parse --short HEAD)-$(date --utc +'%Y%m%d%H%M')"
+
+      - uses: actions/setup-go@v2
+        with:
+          go-version: 1.15
+      - uses: actions/cache@v2
+        with:
+          path: ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-
+      - name: Fetch dependencies
+        run: go get -d -v
+      - name: Test
+        run: go test -v ./...
+      - name: Build binaries
+        env:
+          LDFLAGS: "-X github.com/spinnaker/spin/version.Version=${{ steps.build_variables.outputs.VERSION }}"
+        run: |
+          GOARCH=amd64 GOOS=darwin go build -ldflags "${LDFLAGS}" -o dist/${{ steps.build_variables.outputs.VERSION }}/darwin/amd64/spin .
+          GOARCH=amd64 GOOS=linux go build -ldflags "${LDFLAGS}" -o dist/${{ steps.build_variables.outputs.VERSION }}/linux/amd64/spin .
+          GOARCH=amd64 GOOS=windows go build -ldflags "${LDFLAGS}" -o dist/${{ steps.build_variables.outputs.VERSION }}/windows/amd64/spin.exe .
+          dist/${{ steps.build_variables.outputs.VERSION }}/linux/amd64/spin --version
+
+      - name: Build container image
+        uses: docker/build-push-action@v2
+        with:
+          context: .
+          file: Dockerfile
+          push: false
+          build-args: |
+            "VERSION=${{ steps.build_variables.outputs.VERSION }}"
+          tags: |
+            "${{ env.CONTAINER_REGISTRY }}/${{ steps.build_variables.outputs.REPO }}:latest"
+            "${{ env.CONTAINER_REGISTRY }}/${{ steps.build_variables.outputs.REPO }}:${{ steps.build_variables.outputs.VERSION }}"

.github/workflows/release.yml

@@ -0,0 +1,144 @@
+name: Release
+
+on:
+  push:
+    tags:
+    - "v[0-9]+.[0-9]+.[0-9]+"
+    - "v[0-9]+.[0-9]+.[0-9]+-rc.[0-9]+"
+
+env:
+  CONTAINER_REGISTRY: us-docker.pkg.dev/spinnaker-community/docker
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Ensure code formatting and style is consistent
+      uses: golangci/golangci-lint-action@v2
+      with:
+        version: v1.32
+
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+      - name: Assemble release info
+        id: release_info
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          . .github/workflows/release_info.sh ${{ github.event.repository.full_name }}
+          echo ::set-output name=CHANGELOG::$(echo -e "${CHANGELOG}")
+          echo ::set-output name=SKIP_RELEASE::${SKIP_RELEASE}
+          echo ::set-output name=IS_CANDIDATE::${IS_CANDIDATE}
+          echo ::set-output name=RELEASE_VERSION::${RELEASE_VERSION}
+      - name: Prepare build variables
+        id: build_variables
+        run: |
+          echo ::set-output name=REPO::${GITHUB_REPOSITORY##*/}
+          echo ::set-output name=VERSION::"$(git rev-parse --short HEAD)-$(date --utc +'%Y%m%d%H%M')"
+
+      - uses: actions/setup-go@v2
+        with:
+          go-version: 1.15
+      - uses: actions/cache@v2
+        with:
+          path: ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-
+      - name: Fetch dependencies
+        run: go get -d -v
+      - name: Test
+        run: go test -v ./...
+      - name: Build binaries
+        env:
+          LDFLAGS: "-X github.com/spinnaker/spin/version.Version=${{ steps.release_info.outputs.RELEASE_VERSION }}"
+        run: |
+          GOARCH=amd64 GOOS=darwin go build -ldflags "${LDFLAGS}" -o dist/${{ steps.release_info.outputs.RELEASE_VERSION }}/darwin/amd64/spin .
+          GOARCH=amd64 GOOS=linux go build -ldflags "${LDFLAGS}" -o dist/${{ steps.release_info.outputs.RELEASE_VERSION }}/linux/amd64/spin .
+          GOARCH=amd64 GOOS=windows go build -ldflags "${LDFLAGS}" -o dist/${{ steps.release_info.outputs.RELEASE_VERSION }}/windows/amd64/spin.exe .
+          dist/${{ steps.release_info.outputs.RELEASE_VERSION }}/linux/amd64/spin --version
+      - name: Login to Google Cloud
+        # Only run this on repositories in the 'spinnaker' org, not on forks.
+        if: startsWith(github.repository, 'spinnaker/')
+        uses: 'google-github-actions/auth@v0'
+        # use service account flow defined at: https://github.com/google-github-actions/upload-cloud-storage#authenticating-via-service-account-key-json
+        with:
+          credentials_json: '${{ secrets.GAR_JSON_KEY }}'
+      - name: Upload spin CLI binaries to GCS
+        # Only run this on repositories in the 'spinnaker' org, not on forks.
+        if: startsWith(github.repository, 'spinnaker/')
+        uses: 'google-github-actions/upload-cloud-storage@v0'
+        with:
+          path: 'dist/'
+          destination: 'spinnaker-artifacts/spin'
+          parent: false
+      - name: Determine latest spin CLI version and write to '(MAJOR.MINOR.x-)latest' files
+        run: |
+          gcs_latest="$(curl -s https://storage.googleapis.com/spinnaker-artifacts/spin/latest)"
+          release=${{ steps.release_info.outputs.RELEASE_VERSION }}
+          # SemVer sort 1.22.0 as later than 1.21.1
+          latest="$(printf "%s\n%s" "${gcs_latest}" "${release}" | sort -V | tail -n1)"
+
+          cat <<EOF
+          gcs_latest: $gcs_latest
+          release: $release
+          latest: $latest
+          EOF
+
+          mkdir -p dist/latests
+          echo "$latest" > dist/latests/latest
+
+          # Bump latest version in MAJOR.MINOR
+          major_minor="$(echo "$release" | cut -d '.' -f1-2)"
+          echo "$release" > dist/latests/"${major_minor}.x-latest"
+      - name: Upload latest version reference files
+        # Only run this on repositories in the 'spinnaker' org, not on forks.
+        if: startsWith(github.repository, 'spinnaker/')
+        # See: https://spinnaker.io/docs/setup/other_config/spin/
+        uses: 'google-github-actions/upload-cloud-storage@v0'
+        with:
+          path: 'dist/latests'
+          destination: 'spinnaker-artifacts/spin'
+          parent: false
+
+      - name: Login to GAR
+        # Only run this on repositories in the 'spinnaker' org, not on forks.
+        if: startsWith(github.repository, 'spinnaker/')
+        uses: docker/login-action@v1
+        # use service account flow defined at: https://github.com/docker/login-action#service-account-based-authentication-1
+        with:
+          registry: us-docker.pkg.dev
+          username: _json_key
+          password: ${{ secrets.GAR_JSON_KEY }}
+      - name: Build and publish container image
+        # Only run this on repositories in the 'spinnaker' org, not on forks.
+        if: startsWith(github.repository, 'spinnaker/')
+        uses: docker/build-push-action@v2
+        with:
+          context: .
+          file: Dockerfile.slim
+          push: true
+          build-args: |
+            "VERSION=${{ steps.release_info.outputs.RELEASE_VERSION }}"
+          tags: |
+            "${{ env.CONTAINER_REGISTRY }}/${{ steps.build_variables.outputs.REPO }}:${{ steps.release_info.outputs.RELEASE_VERSION }}"
+            "${{ env.CONTAINER_REGISTRY }}/${{ steps.build_variables.outputs.REPO }}:${{ steps.release_info.outputs.RELEASE_VERSION }}-${{ steps.build_variables.outputs.VERSION }}"
+
+      - name: Create release
+        if: steps.release_info.outputs.SKIP_RELEASE == 'false'
+        uses: actions/create-release@v1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          tag_name: ${{ github.ref }}
+          release_name: ${{ github.event.repository.name }} ${{ github.ref }}
+          body: |
+            ${{ steps.release_info.outputs.CHANGELOG }}
+          draft: false
+          prerelease: ${{ steps.release_info.outputs.IS_CANDIDATE }}

.github/workflows/release_info.sh

@@ -0,0 +1,35 @@
+#!/bin/bash -x
+
+# Only look to the latest release to determine the previous tag -- this allows us to skip unsupported tag formats (like `version-1.0.0`)
+export PREVIOUS_TAG=`curl --silent "https://api.github.com/repos/$1/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/'`
+echo "PREVIOUS_TAG=$PREVIOUS_TAG"
+export NEW_TAG=${GITHUB_REF/refs\/tags\//}
+echo "NEW_TAG=$NEW_TAG"
+export CHANGELOG=`git log $NEW_TAG...$PREVIOUS_TAG --oneline`
+echo "CHANGELOG=$CHANGELOG"
+
+#Format the changelog so it's markdown compatible
+CHANGELOG="${CHANGELOG//$'%'/%25}"
+CHANGELOG="${CHANGELOG//$'\n'/%0A}"
+CHANGELOG="${CHANGELOG//$'\r'/%0D}"
+
+# If the previous release tag is the same as this tag the user likely cut a release (and in the process created a tag), which means we can skip the need to create a release
+export SKIP_RELEASE=`[[ "$PREVIOUS_TAG" = "$NEW_TAG" ]] && echo "true" || echo "false"`
+
+# https://github.com/fsaintjacques/semver-tool/blob/master/src/semver#L5-L14
+NAT='0|[1-9][0-9]*'
+ALPHANUM='[0-9]*[A-Za-z-][0-9A-Za-z-]*'
+IDENT="$NAT|$ALPHANUM"
+FIELD='[0-9A-Za-z-]+'
+SEMVER_REGEX="\
+^[vV]?\
+($NAT)\\.($NAT)\\.($NAT)\
+(\\-(${IDENT})(\\.(${IDENT}))*)?\
+(\\+${FIELD}(\\.${FIELD})*)?$"
+
+# Used in downstream steps to determine if the release should be marked as a "prerelease" and if the build should build candidate release artifacts
+export IS_CANDIDATE=`[[ $NEW_TAG =~ $SEMVER_REGEX && ! -z ${BASH_REMATCH[4]} ]] && echo "true" || echo "false"`
+
+# This is the version string we will pass to the build, trim off leading 'v' if present
+export RELEASE_VERSION=`[[ $NEW_TAG =~ $SEMVER_REGEX ]] && echo "${NEW_TAG:1}" || echo "${NEW_TAG}"`
+echo "RELEASE_VERSION=$RELEASE_VERSION"