Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release: dynamic connector VERSION file #301

Merged
merged 3 commits into from
Jul 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ jobs:
run: |
TAG=$(echo $GITHUB_SHA | head -c7)
echo ::set-output name=tag::${TAG}
VERSION=$(cat base-image/VERSION | tr -d '\n')
echo ::set-output name=version::${VERSION}

- name: Login to GitHub package docker registry
run: |
Expand Down Expand Up @@ -60,7 +62,7 @@ jobs:
context: .
file: base-image/Dockerfile
push: true # See 'if' above
tags: ghcr.io/estuary/base-image:dev,ghcr.io/estuary/base-image:v1
tags: ghcr.io/estuary/base-image:dev,ghcr.io/estuary/base-image:${{ steps.prep.outputs.version }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not seeing why we'd want the version applied to the base image. All of the connectors Dockerfiles that use the base image hard-code the version as :v1. But that's also not making a ton of sense to me, since the build always builds and tags the base image with :local, and only tags it with :v1 on a push to main. So I'd think as it is, that if you make a change to the base image in a PR, then the connector builds won't actually use that change until after it's been merged to main.

Basically, this seems like it was already broken before this PR, but adding the dynamic version to base image definitely doesn't seem like it would help. I'm not confident that I fully understand the purpose of base-image, so take this with a grain of salt. But it seems to me like we could only build the :local tag of the base image and just use that in the connector builds instead of v1.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@psFried

So I'd think as it is, that if you make a change to the base image in a PR, then the connector builds won't actually use that change until after it's been merged to main.

That's true, I don't know what's the best way out of that. If you really want to try something in your PR, you can change your PR to include a change to a connector to use base-image:local or something of the sort.

But it seems to me like we could only build the :local tag of the base image and just use that in the connector builds instead of v1.

The idea is we want the base-image to be available to 3rd-parties who want to create connectors for Flow as well, because we have certain requirements that might change (think: we added OpenSSH requirements to docker images, base image serves that).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't foresee breaking changes in the base-image to be frequent at all. We will realistically only want to break the base-image if we are making some serious changes to how we run connectors.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My two cents: I think given that we want base-image to be published to ghcr.io for others to use it does make sense to give it a version tag, but it seems confusing and unexpected that when making a change to base-image in a PR the new connectors built by that CI run won't include the change. Could we use the :local tag in our connector builds but still push the version-file-tagged base-image?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like CI builds are failing because the ghcr.io/estuary/base-image:local image built in one job isn't visible to the other matrix job. Cursory googling suggests that you can export the whole Docker image as a file and move that between jobs though (see docker/build-push-action#225 (comment) for instance).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried my best on this, but no luck.

Apparently docker buildx does not pick up local images: docker/buildx#847
This means even if we move the image from the build-base to build-connector job, it doesn't get picked up by buildx. It doesn't even get picked up if you build the base-image in build-connector job, it always tries to fetch the image from remote, and will fail if it doesn't exist.

There is probably one hacky way around this which is spinning up a local registry in the CI job, pushing to that local registry and using that, but that means our Dockerfiles will have FROM localhost:5000/base-image:local in them, which makes building them on our local machines cumbersome and hacky and unintuitive.

All in all, I don't really like this solution. I'd say let's try to live with this until buildx supports a way of using local images in its builds.

Copy link
Member

@willdonnelly willdonnelly Jul 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think we've at least established that this is complex enough it shouldn't be tacked on to this change.

(That said, I think it may still be possible to make local builds work via something like ARG BASE_IMAGE=ghcr.io/estuary/base-image:local and then FROM ${BASE_IMAGE} (relevant docs), and then by overriding the BASE_IMAGE variable during our CI builds it could be pointed at another registry)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@willdonnelly that's a good idea, given that it seems to support value substitution just like bash (e.g. ${variable:-default}), we can default to v1 image to allow local builds to run just fine. In this case, we don't even need to move the image, or a local registry, since we do publish the hash-tagged base-image. We can just point to that. I'll look at it tomorrow.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it working 👍🏽

Copy link
Member Author

@mdibaiee mdibaiee Jul 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So now image builds in CI (except the final dev + version build of connectors) use the local base-image


build_connectors:
runs-on: ubuntu-20.04
Expand Down Expand Up @@ -98,6 +100,8 @@ jobs:
run: |
TAG=$(echo $GITHUB_SHA | head -c7)
echo ::set-output name=tag::${TAG}
VERSION=$(cat ${{ matrix.connector }}/VERSION | tr -d '\n')
echo ::set-output name=version::${VERSION}

- name: Download latest Flow release binaries
run: ./fetch-flow.sh
Expand Down Expand Up @@ -155,6 +159,7 @@ jobs:
context: .
file: ${{ matrix.connector }}/Dockerfile
load: true
build-args: BASE_IMAGE=ghcr.io/estuary/base-image:${{ steps.prep.outputs.tag }}
tags: ghcr.io/estuary/${{ matrix.connector }}:local
secrets: |
"rockset_api_key=${{ secrets.ROCKSET_API_KEY }}"
Expand Down Expand Up @@ -212,6 +217,7 @@ jobs:
uses: docker/build-push-action@v2
with:
context: .
build-args: BASE_IMAGE=ghcr.io/estuary/base-image:${{ steps.prep.outputs.tag }}
file: ${{ matrix.connector }}/Dockerfile
push: true
tags: ghcr.io/estuary/${{ matrix.connector }}:${{ steps.prep.outputs.tag }}
Expand All @@ -223,4 +229,4 @@ jobs:
context: .
file: ${{ matrix.connector }}/Dockerfile
push: true # See 'if' above
tags: ghcr.io/estuary/${{ matrix.connector }}:dev,ghcr.io/estuary/${{ matrix.connector }}:v1
tags: ghcr.io/estuary/${{ matrix.connector }}:dev,ghcr.io/estuary/${{ matrix.connector }}:${{ steps.prep.outputs.version }}
1 change: 1 addition & 0 deletions base-image/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions materialize-bigquery/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-bigquery

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion materialize-bigquery/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-bullseye as builder
Expand All @@ -23,7 +25,7 @@ RUN go build -tags nozstd -v -o ./connector ./materialize-bigquery/...

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

RUN apt-get update -y \
&& apt-get install --no-install-recommends -y ca-certificates \
Expand Down
1 change: 1 addition & 0 deletions materialize-bigquery/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions materialize-elasticsearch/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-elasticsearch

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion materialize-elasticsearch/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM --platform=linux/amd64 golang:1.17-bullseye as builder
Expand Down Expand Up @@ -25,7 +27,7 @@ RUN go build -tags nozstd -v -o ./connector ./materialize-elasticsearch

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions materialize-elasticsearch/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions materialize-firebolt/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-firebolt

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion materialize-firebolt/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM --platform=linux/amd64 golang:1.17-bullseye as builder
Expand All @@ -24,7 +26,7 @@ RUN go build -tags nozstd -v -o ./connector ./materialize-firebolt

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions materialize-firebolt/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions materialize-google-sheets/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-google-sheets

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion materialize-google-sheets/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-bullseye as builder
Expand All @@ -24,7 +26,7 @@ RUN go build -tags nozstd -v -o ./connector ./materialize-google-sheets

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions materialize-google-sheets/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions materialize-postgres/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-postgres

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion materialize-postgres/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM --platform=linux/amd64 golang:1.17-bullseye as builder
Expand All @@ -23,7 +25,7 @@ RUN go build -tags nozstd -v -o ./connector ./materialize-postgres/...

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions materialize-postgres/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions materialize-rockset/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-rockset

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion materialize-rockset/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-bullseye as builder
Expand Down Expand Up @@ -29,7 +31,7 @@ RUN go build -tags nozstd -v -o ./connector ./materialize-rockset/cmd/connector/

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions materialize-rockset/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions materialize-s3-parquet/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-s3-parquet

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion materialize-s3-parquet/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-bullseye as builder
Expand All @@ -23,7 +25,7 @@ RUN go build -tags nozstd -v -o ./connector ./materialize-s3-parquet

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions materialize-s3-parquet/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions materialize-snowflake/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-snowflake

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion materialize-snowflake/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-bullseye as builder
Expand All @@ -23,7 +25,7 @@ RUN go build -tags nozstd -v -o ./connector ./materialize-snowflake/...

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

RUN apt-get update -y \
&& apt-get install --no-install-recommends -y \
Expand Down
1 change: 1 addition & 0 deletions materialize-snowflake/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions materialize-sql/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-sql

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 4 additions & 0 deletions materialize-webhook/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# materialize-webhook

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion materialize-webhook/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-bullseye as builder
Expand All @@ -23,7 +25,7 @@ RUN go build -tags nozstd -v -o ./connector ./materialize-webhook/...

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions materialize-webhook/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions source-gcs/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# source-gcs

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion source-gcs/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-buster as builder
Expand All @@ -23,7 +25,7 @@ RUN go build -o ./connector -v ./source-gcs/...

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions source-gcs/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions source-hello-world/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# source-hello-world

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion source-hello-world/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-buster as builder
Expand All @@ -21,7 +23,7 @@ RUN go build -o ./connector -v ./source-hello-world/...

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions source-hello-world/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions source-http-file/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# source-http-file

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion source-http-file/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-buster as builder
Expand All @@ -23,7 +25,7 @@ RUN go build -o ./connector -v ./source-http-file/...

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions source-http-file/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions source-kafka/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# source-kafka

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion source-kafka/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM rust:1.54-slim-buster as builder
Expand Down Expand Up @@ -33,7 +35,7 @@ RUN touch src/main.rs \

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions source-kafka/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions source-kinesis/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# source-kinesis

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion source-kinesis/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-buster as builder
Expand All @@ -22,7 +24,7 @@ RUN go build -o ./connector -v ./source-kinesis/...

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions source-kinesis/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions source-mysql/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# source-mysql

## v1, 2022-07-27
- Beginning of changelog.
4 changes: 3 additions & 1 deletion source-mysql/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage
################################################################################
FROM golang:1.17-buster as builder
Expand All @@ -23,7 +25,7 @@ RUN go build -o ./connector -v ./source-mysql/...

# Runtime Stage
################################################################################
FROM ghcr.io/estuary/base-image:v1
FROM ${BASE_IMAGE}

WORKDIR /connector
ENV PATH="/connector:$PATH"
Expand Down
1 change: 1 addition & 0 deletions source-mysql/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1
4 changes: 4 additions & 0 deletions source-postgres/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# source-postgres

## v1, 2022-07-27
- Beginning of changelog.
Loading