Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildkite: Add dynamic pipeline generator for conditional skipping #2702

Merged
merged 1 commit into from
Feb 21, 2020

Conversation

tjanez
Copy link
Member

@tjanez tjanez commented Feb 21, 2020

Use Buildkite's dynamic pipelines feature to dynamically choose between Code/Code-skip pipelines depending on whether the build was triggered for a pull request and the list of changes files.

Closes #2573.

@tjanez tjanez force-pushed the tjanez/ci-improvements branch 3 times, most recently from 50a9bdd to 4a66973 Compare February 21, 2020 13:02
@codecov
Copy link

codecov bot commented Feb 21, 2020

Codecov Report

Merging #2702 into master will increase coverage by 0.11%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2702      +/-   ##
==========================================
+ Coverage   63.09%   63.20%   +0.11%     
==========================================
  Files         377      377              
  Lines       35448    35448              
==========================================
+ Hits        22365    22405      +40     
+ Misses      10273    10240      -33     
+ Partials     2810     2803       -7     
Impacted Files Coverage Δ
go/worker/common/host/interface.go 38.46% <0.00%> (-15.39%) ⬇️
go/common/grpc/proxy/proxy.go 58.87% <0.00%> (-5.61%) ⬇️
go/worker/compute/txnscheduler/committee/node.go 62.34% <0.00%> (-2.17%) ⬇️
go/runtime/committee/nodes.go 80.64% <0.00%> (-2.16%) ⬇️
...o/consensus/tendermint/apps/scheduler/scheduler.go 73.07% <0.00%> (-1.65%) ⬇️
go/worker/common/committee/group.go 80.32% <0.00%> (ø) ⬆️
go/worker/compute/merge/committee/node.go 75.87% <0.00%> (+0.49%) ⬆️
go/storage/api/grpc.go 70.56% <0.00%> (+0.66%) ⬆️
go/runtime/transaction/transaction.go 77.10% <0.00%> (+1.20%) ⬆️
go/worker/common/p2p/p2p.go 66.21% <0.00%> (+1.35%) ⬆️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5afa3e6...209b0a3. Read the comment docs.

Copy link
Member

@kostko kostko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also remove buildkite/scripts/skip_if_only_docs_changes.sh (and its use) now?

@tjanez tjanez force-pushed the tjanez/ci-improvements branch from 4a66973 to 308dd81 Compare February 21, 2020 13:17
Use Buildkite's dynamic pipelines feature to dynamically choose between
Code/Code-skip pipelines depending on whether the build was triggered
for a pull request and the list of changes files.
@tjanez tjanez force-pushed the tjanez/ci-improvements branch from 308dd81 to 209b0a3 Compare February 21, 2020 13:20
@tjanez
Copy link
Member Author

tjanez commented Feb 21, 2020

Should we also remove buildkite/scripts/skip_if_only_docs_changes.sh (and its use) now?

Yes, good catch. Done.

@tjanez tjanez merged commit 085f221 into master Feb 21, 2020
@tjanez tjanez deleted the tjanez/ci-improvements branch February 21, 2020 14:04
# are no code-related changes compared to the pull request's base branch.
pr_and_no_code_related_changes() {
# Check if the build was triggered for a pull request.
if [[ -z $BUILDKITE_PULL_REQUEST_BASE_BRANCH ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't skip in this case:

  • we should build on master for PR merge commits, because there are steps that are set to run on master only
  • you'd normally push a branch first and then open a PR for it. that first push wouldn't be built under this rule, because a PR doesn't exist yet

@tjanez
Copy link
Member Author

tjanez commented Feb 24, 2020

can't skip in this case:

  • we should build on master for PR merge commits, because there are steps that are set to run on master only

I'm afraid I don't understand. So, for non-pull-request builds, the full code pipeline will always be run. For an example, see the recent builds for the master branch.

  • you'd normally push a branch first and then open a PR for it. that first push wouldn't be built under this rule, because a PR doesn't exist yet

Yes, as I announced internally, I needed to make a slight Buildkite triggering change:

  • Builds won't be triggered for all branches automatically, just the master and stable/* branches.
  • For other branches, one will need to make a pull request.

This is needed since the dynamic Buildkite pipeline needs to know the base branch and changes from it to decide whether to run the full code pipeline or skip it.

@pro-wh
Copy link
Contributor

pro-wh commented Feb 24, 2020

oh ok then thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Skip some CI steps based on pull request's changes
3 participants