Build OSS branch for deploying to Cloud env #11474

pmossman · 2022-03-28T16:58:12Z

What

Part of #9651

The changes in this PR aim to make it easy for a developer to prepare an OSS project branch for deployment to a Cloud environment.

How

The key changes in this PR are summarized as follows (and each have a corresponding commit):

VERSION is now a buildArg in our Dockerfiles that reference an AirbyteVersion. This buildArg defaults to the current AirbyteVersion, and is updated via our usual semver process. Now that it's a buildArg, we can overwrite the default to reference build artifacts that aren't tied to a specific Airbyte version. This is important for building and referencing branch-specific artifacts
There is now a single source of truth for the project artifact version and the project image tag. These values are available in the project.ext block in the top-level build.gradle, as rootProject.ext.version and rootProject.ext.image_tag.
- ext.version tries to use the VERSION environment variable, and falls back on .env.VERSION if unset.
- ext.image_tag tries to use the VERSION environment variable, and falls back on dev if unset.
- This change, in combination with the buildArg change above, allows us to build and tag branch-specific artifacts, while keeping the existing behavior untouched (ie, building local dev images, and building official Airbyte Release versions for prod)
There is a new Github Action that takes in a branch-specific tag as input, builds the project with that tag as the VERSION env var, publishes artifacts to maven local (for the Cloud workflow to consume), and pushes branch-specific docker images to dockerhub (for Cloud to deploy)
- This action will be called by the Cloud github workflow if an OSS branch input is provided. See the Cloud PR that I'll be putting up soon.
The AirbyteVersion class has a version validation that allows official semver, or dev. This validation has been loosened to also allow versions that contain oss-branch. Without this change, the bootloader won't allow branch-specific versions.

Misc

Before taking on this task, I had virtually zero Gradle or Github Action experience, so please please point out things that I'm doing inefficiently. A lot of this was trial and error and learning on the fly, so I really appreciate all feedback to help cement my learning in this new territory!

The Cloud PR will soon follow, though I think it makes sense to review this PR first!

cgardens

VERY COOL!

This action will be called by the Cloud github workflow if an OSS branch input is provided. See the Cloud PR that I'll be putting up soon.

I'm excited to this part!

I think it is worth making sure we have clear docs for this. You could likely re-purpose a lot of what you already have in the PR description.

Only putting comment because I didn't have time to review the github actions closely. The approach seems very good to me.

cgardens · 2022-03-28T19:46:32Z

airbyte-bootloader/Dockerfile

@@ -1,10 +1,14 @@
 ARG JDK_VERSION=17.0.1
 FROM openjdk:${JDK_VERSION}-slim

+ARG VERSION=0.35.60-alpha


why do we need the arg at all? (not a leading question, i genuinely am unsure of the purpose it serves. would it make sense to always pull the version as an arg that must be passed when we build?)

I think we could definitely go that route if we wanted to- the approach I took here with the defaulting behavior was to preserve existing behavior/dev workflows as much as possible. I wanted things to 'just work' if you run SUB_BUILD=PLATFORM ./gradle build

legit. okay. i think my head is gravitating to a separate refactor. seems like we should be able to have SUB_BUILD=PLATFORM ./gradle build "just work" and have gradle be in charge of passing the version in without having to hard code it in the dockerfile.

that said, we should not block on this!

Totally agreed!

cgardens · 2022-03-28T19:53:01Z

.github/actions/build-and-push-branch/action.yml

+name: "Build and Push Branch"
+description: "Build jars and docker images tagged for a particular branch. Primarily used for running OSS branch code in Cloud."
+inputs:
+  branch_version_tag:


is there a way to auto generate this? if we're just doing oss-branch-<commit hash> could the action just generate it itself?

The Cloud workflow generates this based on the input branch name, and then passes it in to the OSS action - I needed Cloud to be in charge of generating this because the same tag/version is used elsewhere in the cloud workflow, and the Cloud workflow is what initially receives the branch name input

fair. you can make it optional fwiw. so if it is not set it just generates oss-branch-<commit-hash> but if it is set it uses what is passed in. probably not all that important though since it sounds like the only reason we would ever call this action is from the GH action in airbyte-cloud

cgardens · 2022-03-28T19:56:52Z

airbyte-commons/src/main/java/io/airbyte/commons/version/AirbyteVersion.java

@@ -133,7 +134,7 @@ public boolean checkOnlyPatchVersionIsUpdatedComparedTo(final AirbyteVersion ano
  }

  public boolean isDev() {
-    return version.equals(DEV_VERSION);
+    return version.equals(DEV_VERSION) || version.contains(OSS_BRANCH_VERSION_PREFIX);


instead of oss-branch-<whatever> would it makes sense to just do dev-<whatever? i haven't thought through the tradeoffs here. you're adding a new magic word. if it has a significantly different meaning than dev it is worth having the name be separate. if they are the same conceptually then maybe repurposing dev to be either the full branch version or just a prefix seems more intuitive?

I think that's a good idea - we'd still need to change the version.equals(DEV_VERSION) to version.contains(DEV_PREFIX) but that feels better to me than a whole other magic word, as you pointed out.

cgardens

Reviewed the github action. Looks good to me. One additional comment on sharing logic in the action, but not necessary to block as long as we create a follow up issue.

cgardens · 2022-03-28T21:51:27Z

.github/actions/build-and-push-branch/action.yml

+    - name: Build
+      run: VERSION=${{ inputs.branch_version_tag }} SUB_BUILD=PLATFORM ./gradlew build --scan
+      shell: bash
+
+    - name: Publish to Maven Local
+      run: VERSION=${{ inputs.branch_version_tag }} SUB_BUILD=PLATFORM ./gradlew publishToMavenLocal
+      shell: bash
+
+    - name: Login to Docker (on Master)
+      uses: docker/login-action@v1
+      with:
+        username: airbytebot
+        password: ${{ inputs.dockerhub_token }}
+
+    - name: Push Docker Images
+      run: |
+        # push the minimum set of images required for Cloud
+        docker push airbyte/worker:${{ inputs.branch_version_tag }}
+        docker push airbyte/webapp:${{ inputs.branch_version_tag }}
+        docker push airbyte/metrics-reporter:${{ inputs.branch_version_tag }}
+        docker push airbyte/scheduler:${{ inputs.branch_version_tag }}
+      shell: bash


it would be nice if we made sure that we use the same commands here as we do in the "real" build that actually publishes releases. I worry that if they diverge over time then this won't be an accurate reflection of reality.

@davinchia may have some thoughts on how to share it. Okay if we don't block on this, but let's make sure we consolidate it shortly after so that we don't stay open to the drift.

I agree it's useful to keep the same docker pushing interface. We could reuse the exact same command as https://github.com/airbytehq/airbyte/blob/master/tools/bin/release_version.sh#L43.

I do think pushing unneeded images is wasteful so I'm leaning towards having a docker-compose-cloud-build.yaml file with a slimmed down image set and using the docker-compose command here.

davinchia · 2022-03-29T13:11:21Z

build.gradle

+// These values are the same for building an specific Airbyte release or branch via the 'VERSION' environment variable.
+// For local development builds, the 'VERSION' environment variable is unset, and built images are tagged with 'dev'.
+ext {
+    version = System.getenv("VERSION") ?: env.VERSION


can we document this somewhere? maybe this https://docs.airbyte.com/contributing-to-airbyte/developing-locally?

.github/actions/build-and-push-branch/action.yml

davinchia

Nice work!

Couple of non-blocking suggestions from me.

supertopher

This looks fantastic, especially given that these are new technologies you dove into. I think Charles and Davin covered the all the nits I saw already. Looks great.

and @cgardens and @davinchia Where if you gif approval game friends?

…ages that are pushed by oss branch action

…v-<commit_hash>

supertopher · 2022-04-05T20:31:53Z

@pmossman ping me (on slack please) if you need another look on any of this work

This reverts commit 189efe7.

This reverts commit 55e3c1e.

* Revert "Revert "Build OSS branch for deploying to Cloud env (#11474)"" This reverts commit 55e3c1e. * add action to get dev branch tag to OSS project instead of doing it in cloud * remove dev branch version action, going to do this in cloud afterall

pmossman requested review from jrhizor, supertopher, cgardens and davinchia March 28, 2022 16:58

pmossman requested a review from a team as a code owner March 28, 2022 16:58

github-actions bot added area/documentation Improvements or additions to documentation area/frontend area/platform issues related to the platform area/scheduler area/server area/worker Related to worker labels Mar 28, 2022

pmossman force-pushed the parker/deploy-oss-to-cloud-env branch from de06836 to 8d5c1a2 Compare March 28, 2022 19:40

pmossman temporarily deployed to more-secrets March 28, 2022 19:42 Inactive

pmossman temporarily deployed to more-secrets March 28, 2022 19:43 Inactive

cgardens reviewed Mar 28, 2022

View reviewed changes

cgardens approved these changes Mar 28, 2022

View reviewed changes

pmossman force-pushed the parker/deploy-oss-to-cloud-env branch from 8d5c1a2 to 82efa83 Compare March 28, 2022 23:53

pmossman temporarily deployed to more-secrets March 28, 2022 23:55 Inactive

pmossman temporarily deployed to more-secrets March 28, 2022 23:56 Inactive

davinchia reviewed Mar 29, 2022

View reviewed changes

pmossman force-pushed the parker/deploy-oss-to-cloud-env branch from 82efa83 to bc9d5a2 Compare March 29, 2022 15:43

pmossman temporarily deployed to more-secrets March 29, 2022 15:46 Inactive

davinchia reviewed Mar 29, 2022

View reviewed changes

.github/actions/build-and-push-branch/action.yml Outdated Show resolved Hide resolved

davinchia approved these changes Mar 29, 2022

View reviewed changes

pmossman force-pushed the parker/deploy-oss-to-cloud-env branch from c088026 to 7cfa3d9 Compare March 30, 2022 16:26

pmossman temporarily deployed to more-secrets March 30, 2022 16:29 Inactive

supertopher approved these changes Mar 30, 2022

View reviewed changes

pmossman temporarily deployed to more-secrets March 30, 2022 16:51 Inactive

pmossman added 6 commits April 5, 2022 09:17

add docker-compose-cloud.build.yaml to define minimum set of cloud im…

740f05b

…ages that are pushed by oss branch action

update local dev docs to describe optional usage of VERSION env var

d64d9d4

make branch_version_tag input optional, if not provided, generates de…

165a2cf

…v-<commit_hash>

fix typo

6fcee4b

fix missed merge conflict

63b8b2d

update docker docs

d0d2d43

pmossman force-pushed the parker/deploy-oss-to-cloud-env branch from 0d504c4 to d0d2d43 Compare April 5, 2022 16:18

pmossman temporarily deployed to more-secrets April 5, 2022 16:20 Inactive

update integrationRunner isDev check

0c3712a

pmossman temporarily deployed to more-secrets April 5, 2022 19:46 Inactive

pmossman merged commit 189efe7 into master Apr 5, 2022

pmossman deleted the parker/deploy-oss-to-cloud-env branch April 5, 2022 22:06

lmossman added a commit that referenced this pull request Apr 5, 2022

Revert "Build OSS branch for deploying to Cloud env (#11474)"

55e3c1e

This reverts commit 189efe7.

pmossman added a commit that referenced this pull request Apr 5, 2022

Revert "Revert "Build OSS branch for deploying to Cloud env (#11474)""

8b1ef71

This reverts commit 55e3c1e.

pmossman added a commit that referenced this pull request Apr 6, 2022

Revert "Revert "Build OSS branch for deploying to Cloud env (#11474)""

75ef853

This reverts commit 55e3c1e.

pmossman mentioned this pull request Apr 7, 2022

Un-Revert OSS branch build for Cloud workflow #11808

Merged

This was referenced Apr 8, 2022

Bump Airbyte version from 0.35.65-alpha to 0.35.66-alpha #11839

Closed

Bump Airbyte version from 0.35.65-alpha to 0.35.66-alpha #11851

Closed

pmossman added a commit that referenced this pull request Apr 8, 2022

Revert "Revert "Build OSS branch for deploying to Cloud env (#11474)""

846c31e

This reverts commit 55e3c1e.

This was referenced Apr 8, 2022

Bump Airbyte version from 0.35.65-alpha to 0.35.66-alpha #11862

Closed

Bump Airbyte version from 0.35.65-alpha to 0.35.66-alpha #11920

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build OSS branch for deploying to Cloud env #11474

Build OSS branch for deploying to Cloud env #11474

pmossman commented Mar 28, 2022

cgardens left a comment

cgardens Mar 28, 2022

pmossman Mar 28, 2022

cgardens Mar 28, 2022

pmossman Mar 28, 2022

cgardens Mar 28, 2022

pmossman Mar 28, 2022

cgardens Mar 28, 2022

cgardens Mar 28, 2022

pmossman Mar 28, 2022

cgardens Mar 28, 2022

cgardens left a comment

cgardens Mar 28, 2022

davinchia Mar 29, 2022

davinchia Mar 29, 2022

davinchia Mar 29, 2022

davinchia left a comment

supertopher left a comment

supertopher commented Apr 5, 2022

Build OSS branch for deploying to Cloud env #11474

Build OSS branch for deploying to Cloud env #11474

Conversation

pmossman commented Mar 28, 2022

What

How

Recommended reading order

Misc

cgardens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgardens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davinchia left a comment

Choose a reason for hiding this comment

supertopher left a comment

Choose a reason for hiding this comment

supertopher commented Apr 5, 2022