infra: Publish nightly releases of the Router from `dev` source #2409

abernix · 2023-01-16T14:44:22Z

Today, when we land features and bug-fixes on dev, they require us cutting a full release before they can be tested by interested/affected parties.

Thanks to work done in #2202 and #2323, we now have just enough automation in our release pipeline to take care of some of the necessary work for building these releases.

With some follow-up work, we can have these triggered through:

Manual triggers through CircleCI
Scheduled pipeline runs in CircleCI

This work touches on ideas from #229 and #242, though neither of those issues are directly "build nightly releases", though I did suggest it in my comments on one of them.

Overall, this creates Router versions of the format v0.0.0-nightly+YYYYMMDD.COMMIT_HASH, which look roughly like router-v0.0.0-nightly+20220102.abcd1234-x86_64-unknown-linux-gnu.tar.gz as binaries which we can attach to non-latest releases on GitHub Releases.

Overall Plan / Follow-ups

I haven't executed this full plan yet, but the near-term follow-ups to this are:

Make these nightly releases available as pre-releases on GitHub Releases
Publish the Docker containers to our GitHub Container Registry repo
Publish nightly builds as Cargo crates to our apollo-router crate

These are still up for discussions.

Checklist

Complete the checklist (and note appropriate exceptions) before a final PR is raised.

Changes are compatible[^1]
Documentation[^2] completed
The documentation is definitely not finished, but that's covered in Release automation improvements #2261
Performance impact assessed and acceptable
Tests added and passing[^3]
- ~~Unit Tests~~
- ~~Integration Tests~~
- Manual Tests

Exceptions
As with most CI-based workflows, testing can be a bit challenging. Overall, this can be tested locally by emulating the steps in the CircleCI file:

Being in a clean Git tree state with the PR checked out.
Running cargo xtask release prepare nightly
mkdir -p artifacts
cargo xtask package --output artifacts/

This will need to be documented as part of #2261

While I once started using GitHub Milestones to try and actually convey what PR landed in which released version of the Router, GitHub now does that automatically with the advent of the "GitHub Releases" feature which automatically puts a green callout at the bottom of a merged PR noting which version of Apollo Router it first was published in. As of today, the Milestones are actually being used _after_ we do a release, which is less helpful and duplicative work. It also has a lot of overhead that required a lot of code to maintain. This removes it, as discussed directly with the core team.

Today, when we land features and bug-fixes on `dev`, they require us cutting a full release before they can be tested by interested/affected parties. Thanks to work done in #2202 and #2323, we now have just enough automation in our release pipeline to take care of some of the necessary work for building these releases. With some follow-up work, we can have these triggered through: - [Manual triggers] through CircleCI - [Scheduled pipeline] runs in CircleCI [Scheduled pipeline]: https://circleci.com/docs/scheduled-pipelines/ [Manual triggers]: https://circleci.com/docs/triggers-overview/#run-a-pipeline-from-the-circleci-web-app This work touches on ideas from #229 and #242, though neither of those issues are _directly_ "build nightly releases", though I did suggest it in my comments on one of them.

abernix · 2023-01-16T14:59:50Z

.circleci/config.yml

+            parameters:
+              platform:
+                [macos_build, windows_build, amd_linux_build, arm_linux_build]
+


Note that this doesn't currently actually publish anything to GitHub Releases like normal releases. Assuming these jobs don't start failing immediately, my next steps are to introduce a step that saves these to CircleCI artifacts:

- store_artifacts: path: ./artifacts

Roughly documented here.

abernix · 2023-01-16T15:05:12Z

.circleci/config.yml

+    when:
+      not: << pipeline.parameters.nightly >>


Mildly frustratingly, this is a requirement of the way that CircleCI invokes workflows, so this will have to be present on each new workflow we may potentially add here in the future.

abernix · 2023-01-16T16:15:33Z

.circleci/config.yml

-      - run:
-          name: Initialize submodules
-          command: git submodule update --recursive --init


This is actually not related to this PR, but should no longer be necessary after #1856

…ilds

abernix · 2023-01-16T16:33:09Z

@abernix, please add a /NEXT_CHANGELOG.md entry for this PR to the appropriate section!

I'm going to forego the changelog until I actually test that this works.

abernix · 2023-01-17T08:15:45Z

xtask/src/commands/release.rs

+                replace_in_file!(
+                    "./apollo-router/Cargo.toml",
+                    r#"^(?P<existingVersion>version\s*=\s*)"[^"]+""#,
+                    format!(
+                        "${{existingVersion}}\"0.0.0-nightly.{}+{}\"",
+                        Utc::now().format("%Y%m%d"),
+                        head_commit
+                    )
+                );


This deviates from the pattern used in the other arms of the match in the way which it assigns the version. It does that because cargo set-version from cargo edit doesn't ever let you set the version lower than the current version.

I considered an approach where we instead stuck the nightly on the existing version number, but it became fragile depending on the preid (e.g., alpha, beta) and it seemed most reasonable (even in terms of defining breaking changes in the spirit of semantic versioning) that 0.0.0 was a more sensical number to use on nightlies which seemed compatible with tooling (and just thought provoking enough to humans for them to consider its meaning).

These artifacts will only survive for 30 days, but they're immediately useful for testing the Nightly builds prior to actually publishing them.

abernix · 2023-01-17T18:23:26Z

The test failures are spurious and unrelated (see #2414, which I opened)

Overall, I just tested this on this PR and this appears to work at surface level! See the Artifacts tab of this PR's build.

.circleci/config.yml

This should probably be an xtask, but our dependency stuff is a bit haywire in here already. (E.g., installing everything in CircleCI YAML instead of with scripts in `xtask`). This took a little bit of extra work because our CircleCI didn't support installing the extra tools on Windows.

…sary For example, not needed for nightly releases!

It uses its own escape hatch to force use of the CLI, much in the same way that Cargo does. This is critical for Windows. Ref: EmbarkStudios/cargo-deny#420

Copy-pasta fail!

…ilds

This is a small addition to #242 which, when coupled with the already-landed #2409, polishes it off nicely by letting us be notified in advance if a nightly build —which is nearly the entire release process — has succeeded or, more importantly, failed. Without this, that functionality is less obvious and too easily missed. Resolves #2409

abernix added 3 commits January 16, 2023 13:38

Put back removed imports that were actually necessary

be9d34c

abernix marked this pull request as ready for review January 16, 2023 14:55

abernix commented Jan 16, 2023

View reviewed changes

abernix self-assigned this Jan 16, 2023

Base automatically changed from abernix/remove-milestones to dev January 16, 2023 16:12

abernix commented Jan 16, 2023

View reviewed changes

abernix added 2 commits January 16, 2023 16:16

Merge branch 'abernix/remove-milestones' into abernix/add-nightly-builds

35b3a6f

Merge remote-tracking branch 'origin/dev' into abernix/add-nightly-bu…

cb2325b

…ilds

This comment was marked as resolved.

Sign in to view

abernix commented Jan 17, 2023

View reviewed changes

abernix added 2 commits January 17, 2023 08:50

Start storing the build artifacts from artifacts/ on CircleCI

9722eb2

These artifacts will only survive for 30 days, but they're immediately useful for testing the Nightly builds prior to actually publishing them.

Prefix seems corrupted and unnecessary so removing it.

f278e5e

abernix requested review from BrynCooke, garypen and bnjjj January 18, 2023 09:18

bnjjj approved these changes Jan 18, 2023

View reviewed changes

Add instructions to the RELEASE_CHECKLIST.md on how to build a nightly

2617501

abernix commented Jan 19, 2023

View reviewed changes

.circleci/config.yml Show resolved Hide resolved

abernix added 4 commits January 19, 2023 16:26

Update .circleci/config.yml

d6339db

Move nightly parameter to correct place.

8b98340

Helm isn't required for nightlies so remove that UX help.

9d413b4

Also skip helm-docs

04ca9c2

abernix force-pushed the abernix/add-nightly-builds branch from 9f738c3 to dfc553f Compare January 19, 2023 20:19

abernix added 2 commits January 19, 2023 20:46

Relax the rule that GITHUB_TOKEN be in the environment when unneces…

aae001e

…sary For example, not needed for nightly releases!

Adapt deny.toml to use git-fetch-with-cli.

e92a9a7

It uses its own escape hatch to force use of the CLI, much in the same way that Cargo does. This is critical for Windows. Ref: EmbarkStudios/cargo-deny#420

abernix force-pushed the abernix/add-nightly-builds branch from dfc553f to e92a9a7 Compare January 19, 2023 21:49

abernix added 2 commits January 19, 2023 21:54

Save to the same cache key that we're restoring from, on Windows.

b0f9c13

Copy-pasta fail!

Merge remote-tracking branch 'origin/dev' into abernix/add-nightly-bu…

68e4fb8

…ilds

abernix enabled auto-merge (squash) January 20, 2023 09:54

abernix merged commit 05ef1d8 into dev Jan 20, 2023

abernix deleted the abernix/add-nightly-builds branch January 20, 2023 10:07

This was referenced Mar 20, 2023

chore: Nightly Release CI Slack notifications #2798

Merged

CI: Add a release pipeline dry run on main #242

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infra: Publish nightly releases of the Router from `dev` source #2409

infra: Publish nightly releases of the Router from `dev` source #2409

abernix commented Jan 16, 2023 •

edited

Loading

abernix Jan 16, 2023

abernix Jan 16, 2023

abernix Jan 16, 2023

This comment was marked as resolved.

abernix commented Jan 16, 2023

abernix Jan 17, 2023

abernix commented Jan 17, 2023

infra: Publish nightly releases of the Router from dev source #2409

infra: Publish nightly releases of the Router from dev source #2409

Conversation

abernix commented Jan 16, 2023 • edited Loading

Overall Plan / Follow-ups

Checklist

abernix Jan 16, 2023

Choose a reason for hiding this comment

abernix Jan 16, 2023

Choose a reason for hiding this comment

abernix Jan 16, 2023

Choose a reason for hiding this comment

This comment was marked as resolved.

abernix commented Jan 16, 2023

abernix Jan 17, 2023

Choose a reason for hiding this comment

abernix commented Jan 17, 2023

infra: Publish nightly releases of the Router from `dev` source #2409

infra: Publish nightly releases of the Router from `dev` source #2409

abernix commented Jan 16, 2023 •

edited

Loading