Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] Migrate branch consistency and protection checks to Buildkite #101646

Merged
merged 2 commits into from
Nov 2, 2023
Merged

[ci] Migrate branch consistency and protection checks to Buildkite #101646

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5ca3f5f
[ci] Migrate branch consistency and protection checks to Buildkite
brianseeders Oct 31, 2023
3932fdf
MAX_WORKERS should not be able to be 0
brianseeders Oct 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .buildkite/pipelines/intake.template.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,23 @@ steps:
agents:
provider: gcp
image: family/elasticsearch-ubuntu-2004
machineType: custom-32-98304
machineType: n1-standard-32
buildDirectory: /dev/shm/bk
- label: part2
command: .ci/scripts/run-gradle.sh -Dbwc.checkout.align=true -Dorg.elasticsearch.build.cache.push=true -Dignore.tests.seed -Dscan.capture-task-input-files checkPart2
timeout_in_minutes: 300
agents:
provider: gcp
image: family/elasticsearch-ubuntu-2004
machineType: custom-32-98304
machineType: n1-standard-32
buildDirectory: /dev/shm/bk
- label: part3
command: .ci/scripts/run-gradle.sh -Dbwc.checkout.align=true -Dorg.elasticsearch.build.cache.push=true -Dignore.tests.seed -Dscan.capture-task-input-files checkPart3
timeout_in_minutes: 300
agents:
provider: gcp
image: family/elasticsearch-ubuntu-2004
machineType: custom-32-98304
machineType: n1-standard-32
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we update the machine type on the BWC jobs as well? I just noticed one fail in intake today with a killed test worker.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can do that separately, I only changed this one in this branch because it's actually already this machine type, it's just not reflected in the template

buildDirectory: /dev/shm/bk
- group: bwc-snapshots
steps:
Expand Down
14 changes: 12 additions & 2 deletions .buildkite/pipelines/periodic.template.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ steps:
agents:
provider: gcp
image: family/elasticsearch-ubuntu-2004
machineType: custom-32-98304
machineType: n1-standard-32
buildDirectory: /dev/shm/bk
env:
ES_RUNTIME_JAVA: "{{matrix.ES_RUNTIME_JAVA}}"
Expand Down Expand Up @@ -82,7 +82,7 @@ steps:
agents:
provider: gcp
image: family/elasticsearch-ubuntu-2004
machineType: custom-32-98304
machineType: n1-standard-32
buildDirectory: /dev/shm/bk
env:
ES_RUNTIME_JAVA: "{{matrix.ES_RUNTIME_JAVA}}"
Expand Down Expand Up @@ -180,3 +180,13 @@ steps:
image: family/elasticsearch-ubuntu-2004
machineType: n2-standard-8
buildDirectory: /dev/shm/bk
- label: Check branch consistency
command: .ci/scripts/run-gradle.sh branchConsistency
timeout_in_minutes: 15
agents:
provider: gcp
image: family/elasticsearch-ubuntu-2004
machineType: n2-standard-2
- label: Check branch protection rules
command: .buildkite/scripts/branch-protection.sh
timeout_in_minutes: 5
10 changes: 10 additions & 0 deletions .buildkite/pipelines/periodic.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1241,3 +1241,13 @@ steps:
image: family/elasticsearch-ubuntu-2004
machineType: n2-standard-8
buildDirectory: /dev/shm/bk
- label: Check branch consistency
command: .ci/scripts/run-gradle.sh branchConsistency
timeout_in_minutes: 15
agents:
provider: gcp
image: family/elasticsearch-ubuntu-2004
machineType: n2-standard-2
- label: Check branch protection rules
command: .buildkite/scripts/branch-protection.sh
timeout_in_minutes: 5
10 changes: 10 additions & 0 deletions .buildkite/scripts/branch-protection.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/bash

set -euo pipefail

STATUS=$(curl -s "https://api.github.com/repos/elastic/elasticsearch/branches/$BUILDKITE_BRANCH" | jq '.protected')
echo "Branch $BUILDKITE_BRANCH protection status is: $STATUS"
if [[ "$STATUS" == "false" ]]; then
echo "Development branch $BUILDKITE_BRANCH is not set as protected in GitHub but should be."
exit 1
fi
8 changes: 4 additions & 4 deletions .ci/jobs.t/elastic+elasticsearch+branch-consistency.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
- job:
name: elastic+elasticsearch+%BRANCH%+branch-consistency
display-name: "elastic / elasticsearch # %BRANCH% - branch consistency"
description: Testing of the Elasticsearch %BRANCH% branch version consistency.
triggers:
- timed: "H 7 * * *"
description: "This job has been migrated to Buildkite.\n"
disabled: true
triggers: []
builders:
- inject:
properties-file: '.ci/java-versions.properties'
properties-file: ".ci/java-versions.properties"
properties-content: |
JAVA_HOME=$HOME/.java/$ES_BUILD_JAVA
- shell: |
Expand Down
6 changes: 3 additions & 3 deletions .ci/jobs.t/elastic+elasticsearch+branch-protection.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
- job:
name: elastic+elasticsearch+%BRANCH%+branch-protection
display-name: "elastic / elasticsearch # %BRANCH% - branch protection"
description: Elasticsearch %BRANCH% branch protection.
description: "This job has been migrated to Buildkite.\n"
disabled: true
node: master
triggers:
- timed: "H 7 * * *"
triggers: []
scm: []
parameters: []
builders:
Expand Down
5 changes: 5 additions & 0 deletions .ci/scripts/run-gradle.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,10 @@ if ! uname -a | grep -q MING; then
export GLIBC_VERSION=$(ldd --version | grep '^ldd' | sed 's/.* \([1-9]\.[0-9]*\).*/\1/')
fi

# Running on 2-core machines without ramdisk can make this value be 0
if [[ "$MAX_WORKERS" == "0" ]]; then
MAX_WORKERS=1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just make it the number of CPUs then in that case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I think it only ends up as 0 if the number of CPUs is 1 or 2... I was thinking that on a 2-CPU machine, there might not really be enough resources to effectively run two tasks in parallel, what do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can leave it at one and tweak later.

fi

set -e
$GRADLEW -S --max-workers=$MAX_WORKERS $@
Loading