Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildkite: Add dynamic pipeline generator for conditional skipping #2702

Merged
merged 1 commit into from
Feb 21, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .buildkite/code-skip.pipeline.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
##
# Code-skip pipeline
##
#
# Buildkite pipeline for skipping running code-related linters and tests.
#

steps:
- label: No code-related changes detected, skipping Code pipeline
command: exit 0
19 changes: 7 additions & 12 deletions .buildkite/pipeline.yml → .buildkite/code.pipeline.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
################
# Build pipeline
################
##
# Code pipeline
##
#
# Main Buildkite pipeline for running code-related linters and tests.
#

docker_plugin_default_config: &docker_plugin_default_config
image: "oasislabs/testing:0.3.0"
always_pull: true
Expand Down Expand Up @@ -90,7 +94,6 @@ steps:
# .buildkite/scripts/download_utils.sh.
- label: Build Go node
command:
- . .buildkite/scripts/skip_if_only_docs_changes.sh
- .buildkite/go/build.sh

# Upload the built artifacts.
Expand All @@ -109,7 +112,6 @@ steps:
# .buildkite/scripts/download_utils.sh.
- label: Build Rust runtime loader
command:
- . .buildkite/scripts/skip_if_only_docs_changes.sh
- .buildkite/rust/build_generic.sh /workdir -p oasis-core-runtime-loader
- .buildkite/rust/build_generic.sh /workdir -p test-long-term-client
- .buildkite/rust/build_generic.sh /workdir -p simple-keyvalue-client
Expand All @@ -136,7 +138,6 @@ steps:
# .buildkite/rust/test_runtime_and_gateway.sh and .buildkite/scripts/download_utils.sh.
- label: Build key manager runtime
command:
- . .buildkite/scripts/skip_if_only_docs_changes.sh
- .buildkite/rust/build_runtime.sh keymanager-runtime
- .buildkite/rust/build_runtime.sh tests/runtimes/simple-keyvalue

Expand All @@ -157,7 +158,6 @@ steps:
###########
- label: Test Rust crates
command:
- . .buildkite/scripts/skip_if_only_docs_changes.sh
# Build storage interoperability test helpers first.
- make build-helpers
- export OASIS_STORAGE_PROTOCOL_SERVER_BINARY=$(realpath go/storage/mkvs/urkel/interop/urkel-test-helpers)
Expand All @@ -177,7 +177,6 @@ steps:
- label: Test fuzz builds
branches: master
command:
- . .buildkite/scripts/skip_if_only_docs_changes.sh
# TODO: Consider making this a part of the development Docker image.
- go get -u github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build
- make -C go build-fuzz
Expand All @@ -189,7 +188,6 @@ steps:
#####################################
- label: Test Go node
command:
- . .buildkite/scripts/skip_if_only_docs_changes.sh
- .buildkite/go/test_and_coverage.sh
artifact_paths:
- coverage-misc.txt
Expand All @@ -205,7 +203,6 @@ steps:
parallelism: 6
timeout_in_minutes: 9
command:
- . .buildkite/scripts/skip_if_only_docs_changes.sh
- .buildkite/scripts/download_e2e_test_artifacts.sh
- .buildkite/scripts/test_e2e.sh
artifact_paths:
Expand All @@ -226,7 +223,6 @@ steps:
parallelism: 5
timeout_in_minutes: 19
command:
- . .buildkite/scripts/skip_if_only_docs_changes.sh
- .buildkite/scripts/download_e2e_test_artifacts.sh
- .buildkite/scripts/test_e2e.sh
artifact_paths:
Expand Down Expand Up @@ -281,7 +277,6 @@ steps:

- label: "Merge and upload coverage"
command:
- . .buildkite/scripts/skip_if_only_docs_changes.sh
- .buildkite/scripts/merge_coverage.sh
- .buildkite/scripts/upload_coverage.sh
artifact_paths:
Expand Down
45 changes: 45 additions & 0 deletions .buildkite/pipeline.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/bin/bash
##
# Dynamic Buildkite pipeline generator.
##
#
# It outputs valid Buildkite pipeline in YAML format.
#
# To use it, define the following Steps under your Buildkite's Pipeline Settings:
#
# steps:
# - command: .buildkite/pipeline.sh | buildkite-agent pipeline upload
# label: ":pipeline: Upload"
#
# For more details, see:
# https://buildkite.com/docs/pipelines/defining-steps#dynamic-pipelines.
#

set -eux

# Helper that ensures the build is triggered for a pull request and that there
# are no code-related changes compared to the pull request's base branch.
pr_and_no_code_related_changes() {
# Check if the build was triggered for a pull request.
if [[ -z $BUILDKITE_PULL_REQUEST_BASE_BRANCH ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't skip in this case:

  • we should build on master for PR merge commits, because there are steps that are set to run on master only
  • you'd normally push a branch first and then open a PR for it. that first push wouldn't be built under this rule, because a PR doesn't exist yet

return 1
fi
# Get the list of changes files, excluding changes unrelated to code.
# NOTE: The exclude patterns below use git's pathspec syntax:
# https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddefpathspecapathspec.
git diff --name-only --exit-code "refs/remotes/origin/$BUILDKITE_PULL_REQUEST_BASE_BRANCH.." 1>&2 -- \
':(exclude)*.md' \
':(exclude).changelog/' \
':(exclude).github/' \
':(exclude).gitlint' \
':(exclude).markdownlint.yml' \
':(exclude).punch_config.py' \
':(exclude)docs/' \
':(exclude)towncrier.toml'
}

if pr_and_no_code_related_changes; then
cat .buildkite/code-skip.pipeline.yml
else
cat .buildkite/code.pipeline.yml
fi
23 changes: 0 additions & 23 deletions .buildkite/scripts/skip_if_only_docs_changes.sh

This file was deleted.

6 changes: 6 additions & 0 deletions .changelog/2702.internal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
ci: Skip some steps for non-code changes

When one makes a pull request that e.g. only adds documentation or
assembles the Change Log from fragments, all the *heavy* Buildkite
pipeline steps (e.g. Go/Rust building, Go tests, E2E tests) should be
skipped.